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BROWSING ELECTRONICALLY-ACCESSIBLE RESOURCES 

Copyright Notice 

This patent document contains material subject to copyright protection. The 
5 copyright owner has no objection to the reproduction of this patent document or any 
related materials in the files of the Patent Office, but otherwise reserves all copyright 
whatsoever. 

Field of Invention 

The present invention relates to a method and apparatus for selecting a description 
10 or part of a description of a resource. The invention also relates to an apparatus and a 
computer program product for implementing the method. 

Background 

As network connectivity has continued its explosive growth and digital storage has 
become smaller, faster, and less expensive, the quantity of electronically-accessible 

15 resources has increased enormously. So much so that the discovery and location of the 
available resources has become a critical problem. These electronically-accessible 
resources can be digital content (e.g., digital images, video and audio) which may be 
available over the network, web-based resources (e.g., HTML/XML documents) and 
electronic devices (e.g., printers, displays, etc.) In addition, there are electronically- 

20 accessible catalogues of other resources, which may not be electronically accessible (e.g., 
books 7 analog film media, etc.). What is needed is a consistent method of describing 
resources so that location of resources, electronically-accessible or otherwise, can be more 
readily achieved. 

The problems of consistent resource description are twofold. First, there is the 
25 problem of acceptance of a standard (consistent) method of resource description. The 
second problem is related to the generation of descriptions. Often the cost of this process 
is significant. 

The ready adoption of HTML browsers for the purposes for browsing and locating 
textual information indicates that humans like to browse for information. Currently, 
30 however no form of generic browsing of non-textual electronically-accessible resources 
exists. For example, browsing of a library of text documents is typically enabled using an 
HTML browser by providing a table of contents and then linking the relevant HTML 
(textual) documents to anchors in an HTML document. Currently, there is no general way 
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of browsing through a library containing, for example, digital video resources. For such 
browsing to be enabled it is necessary to have a consistent way of describing resources in 
much the same way that HTML provides a consistent way of representing textual 
information. 

5 If a consistent method of describing resources can be achieved then consistent 

methods of selecting resource descriptions from formulated queries can be contemplated. 

Furthermore, it is also possible to develop methods and build devices that facilitate 
resource discovery, understanding and presentation. Resources can be composite items 
involving other resources and can include schedules for presentation, delivery and/or 
10 consumption. Video presentations can therefore be viewed as just another electronically- 
accessible resource and as such can be described in a similar way to items of digital 
content (like images, video and audio). The description of the video presentation 
effectively provides an efficient encoding for the presentation. 

Summary of the Invention 
15 It is an object of the present invention to ameliorate one or more disadvantages of 

the prior art. 

According to one aspect of the invention, there is provided a method of browsing 
electronically-accessible resources using descriptions of said resources, wherein said 
descriptions of said resources have descriptor components, said descriptor components 

20 having attributes representative of at least two axes of access to the resources and wherein 
said descriptions have links to corresponding said electronically- accessible resources, 
said method comprising the steps of: reading said descriptions; displaying items, wherein 
each item is associated with a corresponding said descriptor component of a said read 
description that has at least one said attribute; and browsing said descriptions of the 

25 resources and their corresponding electronically-accessible resources via said links using 
said displayed items. 

According to another aspect of the invention, there is provided a method of 
searching electronically-accessible resources using descriptions of said resources, wherein 
said descriptions of said resources have descriptor components, said descriptor 
30 components having attributes representative of at least two axes of access to the resources 
and wherein said descriptions have links to corresponding said electronically- accessible 
resources, said method comprising the steps of: reading said descriptions; displaying 
items, wherein each item is associated with a corresponding said descriptor component of 
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a said read description that has at least one said attribute; specifying a query in terms of 
index descriptors; searching said descriptions of the resources using said query; and 
locating said corresponding electronically-accessible resources using said links associated 
with said displayed items, 
5 According to still another aspect of the invention, there is provided a method of 

locating electronically-accessible resources using descriptions of said resources, wherein 
the descriptions of said resources have descriptor components, each said descriptor 
component comprises the association of a feature of a said resource with a representative 
value for that feature, and one or more of said descriptor components including a table of 

10 contents attribute and one or more of said descriptor components including an index 
attribute, and wherein said descriptions have links to corresponding said electronically- 
accessible resources, said method comprising the steps of: reading said descriptions; 
displaying a table of contents containing table of contents items, wherein each table of 
contents item is associated with a corresponding said descriptor component that has a 

15 table of contents attribute; selecting one said displayed table of contents item; displaying 
an index containing index items, wherein each said displayed index item is associated 
with a corresponding said descriptor component that has an index attribute and that is 
associated with the said selected table of contents item; selecting one or more index items 
in the displayed index; specifying a said representative value or values for the one or more 

20 said selected index items; searching said descriptions of the resources for said one or 
more selected index items and their corresponding said specified representative. yalue or 
values; and locating one or more said descriptions of the resources corresponding to said 
one or more selected index items and their corresponding said specified representative 
value or values, 

25 According to still another aspect of the invention, there is provided a method of 

annotating an electronically-accessible resource using a description of said resource, 
wherein the description of said resource has descriptor components, each said descriptor 
component comprises the association of a feature of said resource with a representative 
value for that feature, and one or more of said descriptor components including a table of 

30 contents attribute and one or more of said descriptor components including an index 
attribute, said method comprising the steps of: reading said descriptions; displaying a 
tabic of contents containing table of contents items, wherein each table of contents item is 
associated with a corresponding said descriptor component that ' has a table of contents 
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attribute; selecting one said displayed table of contents item for the annotation; displaying 
an index containing index items, wherein each said displayed index item is associated 
with a corresponding said descriptor component that has an index attribute and that is 
associated with the said selected table of contents item; selecting one said displayed index 
5 item; associating said selected displayed index item with said selected table of contents 
item; choosing a said representative value for the selected index item; and associating said 
chosen representative value with said feature which corresponds to said selected index 
item, wherein said chosen representative value and its corresponding feature provide an 
annotation of the resource. 

10 According to still another aspect of the invention, there is provided apparatus for 

browsing electronically-accessible resources using descriptions of said resources, wherein 
said descriptions of said resources have descriptor components, said descriptor 
components having attributes representative of at least two axes of access to the resources 
and wherein said descriptions have links to corresponding said electronically- accessible 

15 resources, said apparatus comprising: means for reading said descriptions; means for 
displaying items, wherein each item is associated with a corresponding said descriptor 
component of a said read description that has at least one said attribute; and means for 
browsing said descriptions of the resources and their corresponding electronically- 
accessible resources via said links using said displayed items. 

20 According to still another aspect of the invention, there is provided apparatus for 

searching electronically-accessible resources using descriptions of said resources, .wherein 
said descriptions of said resources have descriptor components, said descriptor 
components having attributes representative of at least two axes of access to the resources 
and wherein said descriptions have links to corresponding said electronically- accessible 

25 resources, said apparatus comprising: means for reading said descriptions; means for 
displaying items, wherein each item is associated with a corresponding said descriptor 
component of a said read description that has at least one said attribute; means for 
specifying a query in terms of index descriptors; means for searching said descriptions of 
the resources using said query; and means for locating said corresponding electronically- 

30 accessible resources using said links associated with said displayed items. 

According to still another aspect of the invention, there is provided apparatus for 
locating electronically-accessible resources using descriptions of said resources, wherein 
the descriptions of said resources have descriptor components, each said descriptor 
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component comprises the association of a feature of a said resource with a representative 
value for that feature, and one or more of said descriptor components including a table of 
contents attribute and one or more of said descriptor components including an index 
attribute, and wherein said descriptions have links to corresponding said electronically- 
accessible resources, said apparatus comprising: means for reading said descriptions; 
means for displaying a table of contents containing table of contents items, wherein each 
table of contents item is associated with a corresponding said descriptor component that 
has a table of contents attribute; means for selecting one said displayed table of contents 
item; means for displaying an index containing index items, wherein each said displayed 
index item is associated with a corresponding said descriptor component that has an index 
attribute and that is associated with the said selected table of contents item; means for 
selecting one or more index items in the displayed index; means for specifying a said 
representative value or values for the one or more said selected index items; means for 
searching said descriptions of the resources for said one or more selected index items and 
their corresponding said specified representative value or values; and means for locating 
one or more said descriptions of the resources corresponding to said one or more selected 
index items and their corresponding said specified representative value or values. 

According to still another aspect of the invention, there is provided apparatus for 
annotating an electronically-accessible resource using a description of said resource, 
wherein the description of said resource has descriptor components, each said descriptor 
component comprises the association of a feature of said resource with a representative 
value for that feature, and one or more of said descriptor components including a table of 
contents attribute and one or more of said descriptor components including an index 
attribute, said apparatus comprising: means for reading said descriptions; means for 
displaying a tabic of contents containing table of contents items, wherein each table of 
contents item is associated with a corresponding said descriptor component that has a 
table of contents attribute; means for selecting one said displayed table of contents item 
for the annotation; means for displaying an index containing index items, wherein each 
said displayed index item is associated with a corresponding said descriptor component 
that has an index attribute and that is associated with the said selected table of contents 
item;means for selecting one said displayed index item; means for associating said 
selected displayed index item with said selected table of contents item; means for 
choosing a said representative value for the selected index item; and means for associating 

CFP1595AU IPR34-38_GRP 1490568 t:\HLEOClSRAMPRMPR34-38_GRP lW005<>8.doc:PWM 



-6- 



said chosen representative value with said feature which corresponds to said selected 
index item, wherein said chosen representative value and its, corresponding feature 
provide an annotation of the resource. 

According to still another aspect of the invention, there is provided a computer 
5 readable medium comprising a computer program for browsing electronically-accessible 
resources using descriptions of said resources, wherein said descriptions of said resources 
have descriptor components, said descriptor components having attributes representative 
of at least two axes of access to the resources and wherein said descriptions have links to 
corresponding said electronically- accessible resources, said computer program 

10 comprising: code for reading said descriptions; code for displaying items, wherein 
each item is associated with a corresponding said descriptor component of a said read 
description that has at least one said attribute; and code for browsing said descriptions of 
the resources and their corresponding electronically-accessible resources via said links 
using said displayed items. 

15 According to still another aspect of the invention, there is provided a computer 

readable medium comprising a computer program for searching electronically-accessible 
resources using descriptions of said resources, wherein said descriptions of said resources 
have descriptor components, said descriptor components having attributes representative 
of at least two axes of access to the resources and wherein said descriptions have links to 

20 corresponding said electronically- accessible resources, said computer program 
comprising: code for reading said descriptions; code for displaying items, wherein each 
item is associated with a corresponding said descriptor component of a said read 
description that has at least one said attribute; code for specifying a query in terms of 
index descriptors; code for searching said descriptions of the resources using said query; 

25 and code for locating said corresponding electronically-accessible resources using said 
links associated with said displayed items . 

According to still another aspect of the invention, there is provided a computer 
readable medium comprising a computer program for locating electronically-accessible 
30 resources using descriptions of said resources, wherein the descriptions of said resources 
have descriptor components, each said descriptor component comprises the association of 
a feature of a said resource with a representative value for thai feature, and one or more of 
said descriptor components including a table of contents attribute and one or more of said 
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descriptor components including an index attribute, and wherein said descriptions have 
links to corresponding said electronically- accessible resources, said computer program 
comprising; code for reading said descriptions; code for displaying a table of contents 
containing table of contents items, wherein each table of contents item is associated with 
a corresponding said descriptor component that has a table of contents attribute; code for 
selecting one said displayed table of contents item; code for displaying an index 
containing index items> wherein each said displayed index item is associated with a 
corresponding said descriptor component that has an index attribute and that is associated 
with the said selected table of contents item; code for selecting one or more index items in 
the displayed index; code for specifying a said representative value or values for the one 
or more said selected index items; code for searching said descriptions of the resources 
for said one or more selected index items and their corresponding said specified 
representative value or values; and code for locating one or more said descriptions of the 
resources corresponding to said one or more selected index items and their corresponding 
said specified representative value or values. 

According to still another aspect of the invention, there is provided a computer 
readable medium comprising a computer program for annotating an electronically- 
accessible resource using a description of said resource, wherein the description of said 
resource has descriptor components, each said descriptor component comprises the 
association of a feature of said resource with a representative value for that feature, and 
one or more of said descriptor components including a table of contents attribute ^uad one 
or more of said descriptor components including an index attribute, said computer 
program comprising: code for reading said descriptions; code for displaying a table of 
contents containing table of contents items, wherein each table of contents item is 
associated with a corresponding said descriptor component that has a table of contents 
attribute; code for selecting one said displayed table of contents item for the annotation; 
code for displaying an index containing index items, wherein each said displayed index 
item is associated with a corresponding said descriptor component that has an index 
attribute and that is associated with the said selected table of contents item; code for 
selecting one said displayed index item; code for associating said selected displayed index 
item with said selected table of contents item; code for choosing a said representative 
value for the selected index item; and code for associating said chosen representative 
value with said feature which corresponds to said selected index item, wherein said 
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chosen representative value and its corresponding feature provide an annotation of the 
resource. 

According to still another aspect of the invention, there is provided a method of 
selecting one or more descriptions or one or more descriptor components from a set of 
5 descriptions, wherein said descriptions comprise one or more said descriptor components, 
and each said description of said set of descriptions is associated with a corresponding 
electronically accessible resource, said method comprising the steps of: specifying a 
desired selection of descriptor components; generating a selection rale based on said 
specified descriptor components, wherein said selection rale having a predetermined 

10 pattern and action component, wherein the said predetermined pattern represents a 
specified pattern of descriptor components and the said action specifies the action to be 
performed when a said descriptor component in the descriptions of the set of descriptions 
matches the predetermined pattern; reading said descriptions of the resources; locating 
patterns of descriptor components in descriptions of said set of descriptions in order to 

15 select descriptor components or descriptions hi said set of descriptions having said 
descriptor components which match the predetermined pattern; and performing said 
specified action. 

According to still another aspect of the invention, there is provided apparatus for 
selecting one or more descriptions or one or more descriptor components from a set of 

20 descriptions, wherein said descriptions comprise one or more said descriptor components, 
and each said description of said set of descriptions is associated with a corresponding 
electronically accessible resource, said apparatus comprising: means for specifying a 
desired selection of descriptor components; means for generating a selection rule based on 
said specified descriptor components, wherein said selection rule having a predetermined 

25 pattern and action component, wherein the said predetermined pattern represents a 
specified pattern of descriptor components and the said action specifies the action to be 
performed when a said descriptor component in the descriptions of the set of descriptions 
matches the predetermined pattern; means for reading said descriptions of the resources; 

means for locating patterns of descriptor componenis in descriptions of said set of 

30 descriptions in order to select descriptor components or descriptions in said set of 
descriptions having said descriptor components which match the predetermined pattern; 
and means for performing said specified action. 

CFP1595AU iPR34-36_GRP!490568 I:\ELEC\CISRA\IPR\TPR34-38_GRP] \4905G8.doc :PWM , 



According to still another aspect of the invention, there is provided a computer 
readable medium comprising a computer program for selecting one or more descriptions 
or one or more descriptor components from a set of descriptions, wherein said 
descriptions comprise one or more said descriptor components, and each said description 
5 of said set of descriptions is associated with a corresponding electronically accessible 
resource, said computer program comprising: code for specifying a desired selection of 
descriptor components; code for generating a selection rule based on said specified 
descriptor components, wherein said selection rule having a predetermined pattern and 
action component, wherein the said predetermined pattern represents a specified pattern 
10 of descriptor components and the said action specifies the action to be performed when a 
said descriptor component in the descriptions of the set of descriptions matches the 
predetermined pattern; code for reading said descriptions of the resources; code for 
locating patterns of descriptor components in descriptions of said set of descriptions in 
order to select descriptor components or descriptions in said set of descriptions having 
15 said descriptor components which match the predetermined pattern; and code for 
performing said specified action. 

According to still another aspect of the invention, there is provided a method of 
generating on an output device a presentation based on a predetermined selection of 
resources, said method comprising the steps of: reading a description scheme foT said 
20 presentation, wherein the description scheme for said presentation uses a declarative 
description definition language which contains definitions for descriptor components of 
the description scheme; generating a description of the said presentation using said 
description scheme and said predetermined selection of said resources; and generating on 
said output device the said presentation based on the said description of the said 
25 presentation and the predetermined selected resources. 

According to still another aspect of the invention, there is provided a method of 
generating on an output device a presentation based on a predetermined selection of 
resources, said method comprising the steps of: reading a description scheme for said 
presentation, wherein said description scheme contains definitions for descriptor 
30 components of the description scheme, and each said descriptor component comprises the 
association of a presentation attribute with a representative value for that attribute; 
generating a description of the said presentation using said description scheme and said 
predetermined selection of resources; and generating on said output device the said 
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presentation based on the said generated description of the said presentation, the 
predetermined selected resources, and a set of presentation rules, which rules specify 
characteristics of the style of said generated presentation, wherein said set of presentation 
rules are associated with said description scheme. 

According to still another aspect of the invention, there is provided apparatus for 
generating on an output device a presentation based on a predetermined selection of 
resources, said apparatus comprising: means for reading a description scheme for said 
presentation, wherein the description scheme for said presentation uses a declarative 
description definition language which contains definitions for descriptor components of 
the description scheme; means for generating a description of the said presentation using 
said description scheme and said predetermined selection of said resources; and means 
for generating on said output device the said presentation based on the said description of 
the said presentation and the predetermined selected resources. 

According to still another aspect of the invention, there is provided apparatus for 
generating on an output device a presentation based on a predetermined selection of 
resources, said apparatus comprising: means for reading a description scheme for said 
presentation, wherein said description scheme contains definitions for descriptor 
components of the description scheme, and each said descriptor component comprises the 
association of a presentation attribute with a representative value for that attribute; means 
for generating a description of the said presentation using said description scheme and 
said predetermined selection of resources; and means for generating on said output device 
the said presentation based on the said generated description of the said presentation, the 
predetermined selected resources, and a set of presentation rules, which rules specify 
characteristics of the style of said generated presentation, wherein said set of presentation 
rules are associated with said description scheme. 

According to still another aspect of the invention, there is provided a computer 
readable medium comprising a computer program for generating on an output device a 
presentation based on a predetermined selection of resources, said computer program 
comprising: code for reading a description scheme for said presentation, wherein the 
description scheme for said presentation uses a declarative description definition language 
which contains definitions for descriptor components of the description scheme; 
code for generating a description of the said presentation using said description scheme 
and said predetermined selection of said resources; and code for generating on said output 
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device the said presentation based on the said description of the said presentation and the 
predetermined selected resources. 

According to still another aspect of the invention, there is provided a computer 
readable medium comprising a computer program for generating on an output device a 
5 presentation based on a predetermined selection of resources, said computer program 
comprising; code for reading a description scheme for said presentation, wherein said 
description scheme contains definitions for descriptor components of the description 
scheme, and each said descriptor component comprises the association of a presentation 
attribute with a representative value for that attribute; code for generating a description of 

10 the said presentation using said description scheme and said predetermined selection of 
resources; and code for generating on said output device the said presentation based on 
the said generated description of the said presentation, the predetermined selected 
resources, and a set of presentation rules, which rules specify characteristics of the style of 
said generated presentation, wherein said set of presentation rules are associated with said 

15 description scheme. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Embodiments of the invention are described with reference to the accompanying 
5 drawings, in which: 

Fig. 1 A shows a flow diagram of a method of generating a description of a resource 
in accordance with an embodiment; 

Fig. IB shows a flow diagram of a method of processing a description of a resource 
in accordance with another embodiment; 
10 Fig. 1C shows a flow diagram of a method of encoding a description of a resource 

in accordance with another embodiment; 

Fig. ID shows a flow diagram of a method of decoding an encoded description of a 
resource in accordance with another embodiment; 

Figs, 2 A shows a flow diagram of a prior art method of generating a document 
15 object model; 

Fig 2B show a flow diagram of a method of generating a Description Object Model 
in accordance with another embodiment; 

Fig, 3 shows a UML class diagram showing core elements of the Dynamic 
Description Framework(DDF) data model; 
20 Fig. 4 shows a schematic drawing depicting the processing model of an exemplary 

description according to a DDF; 

Fig. 5 shows a schematic drawing depicting the processing model of another 
exemplary description according to a DDF; 

Fig. 6 shows a schematic drawing depicting the relationship between a description 
25 scheme (Document Type Definition) and descriptions( XML documents); 

Fig. 7A is a flow diagram of a method of generating a description of a resource in 
accordance another embodiment; 

Fig. 7B is a flow diagram of a method of processing a description of a resource in 
accordance another embodiment; 
30 Fig. 8 shows an example of the use of a descriptor handler in generating video 

segment description using camera metadata that is saved to the video; 

Fig. 9 shows an example of the use of descriptor handlers to support a query-by- 
example over remote image databases; 
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Fig. 10 shows an example of the use a descriptor handler for ;encoding/decoding; 

Fig. 1 1 shows an example of descriptor handlers implemented as Java classes; 

Fig, 12 is a flow diagram of a method of extending a description of a resource in 
accordance with another embodiment; 

Fig. 13 is a flow diagram of a method of using rules to add or remove attributes of a 
Description Scheme that are used to control the presentation of a description of a resource 
in accordance with another embodiment; 

Fig. 14 is a flow diagram of a method of selecting one or more descriptions or part 
of one or more descriptions of a resource in accordance with another embodiment; 

Fig. 15 is a flow diagram of a method of translating a description of a resource in 
accordance with another embodiment; 

Fig, 16 shows a schematic diagram Digital Video Browser System in accordance 
with another embodiment; 

Fig. 17 shows an implementation of the Digital Video Browser System in a remote 
handheld device in accordance with another embodiment; 

Fig. 18 shows an alternative implementation of the Digital Video Browser System 
in a remote handheld device in accordance with another embodiment; 

Fig. 19 is a block diagram of a general-purpose computer for implementing any one 
or more said methods; and 

Fig. 20 shows a schematic diagram of an Media Browser System in accordance with 
another embodiment. 

Where reference is made in any one or more of the accompanying drawings to steps 
ancVor features, which have the same reference numerals, those steps and/or features-have 
for the purposes of this description the same function(s) or operation(s), unless the 
contrary intention appears. 

BRIEF DESCRIPTION OF THE APPENDICES 
Embodiments of the invention are also described with reference to the appendices, 
in which: 

Appendix A shows core DDF element definitions; 

Appendix B shows an example description scheme for an Australian Football 
League Game; 

Appendix C shows an example description generated from the description scheme 
in Appendix B; 
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Appendix D shows a digital video resource description scheme; 
Appendix E shows an example description generated from the video description 
scheme in Appendix D; 

Appendix F shows presentation rules for the video description scheme in Appendix 

5 D; 

Appendix G shows a digital video library description scheme; 

Appendix H shows an example description generated from the digital video library 
description scheme in Appendix G; 

Appendix I shows video presentation description scheme; 
10 Appendix J shows an example description generated from the video presentation 

description scheme in Appendix I; and 

Appendix K shows DOM element nodes. 
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1. INTRODUCTION 

For a better understanding of the embodiments, an introduction (Section 1) 
including a brief review of terminology (Section 1.1) is first undertaken, then there is 
provided a discussion of relationships between components of descriptions (Section 1.2), 
5 the DDF (Section 2), the serialisation syntax specification (Section 3), and the DesOM 
API specification (Section 4) used in the embodiments. A more; detailed description of 
the embodiments is then given in Sections 6 to 15. 

Some portions of the detailed descriptions which follow are explicitly or implicitly 
presented in terms of algorithms and symbolic representations of operations on data 
10 within a computer memory. These algorithmic descriptions and representations are the 
means used by those skilled in the data processing arts to most effectively convey the 
substance of their work to others skilled in the art. An algorithm is here, and generally, 
conceived to be a self-consistent sequence of steps leading to a desired result. The steps 
are those requiring physical manipulations of physical quantities. Usually, though not 
15 necessarily, these quantities take the form of electrical or magnetic signals capable of 
being stored, transferred, combined, compared, and otherwise majnipulated. It has proven 
convenient at times, principally for reasons of common usage, to refer to these signals as 
bits, values, elements, symbols, characters, terms, numbers, or the like. 

It should be borne in mind, however, that all of these and: similar terms are to be 
20 associated with the appropriate physical quantities and are merely convenient labels 
applied to these quantities. Unless specifically stated otherwise as apparent from the 
following discussions, it is appreciated that throughout the present invention, discussions 
utilising terms such as "processing", "computing", "generating 7 ', "creating", "operating" 
"communicating", "rendering", "providing", and "linking" or the like, refer to the action 
25 and processes of a computer system, or similar electronic computing device, that 
manipulates and transforms data represented as physical (electronic) quantities within the 
computer system's registers and memories into other data similarly represented as physical 
quantities within the computer system memories or registers or other such information 
storage, transmission or display devices. 
30 The present invention also relates to apparatus for performing the operations herein. 

This apparatus may be specially constructed for the required purposes, or it may comprise 
a general purpose computer selectively activated or reconfigured by a computer program 
stored in the computer. The algorithms and displays presented herein are not inherently 
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related to any particular computer or other apparatus. Various general purpose machines 
may be used with programs in accordance with the teachings herein, or it may prove 
convenient to construct more specialised apparatus to perform the required method steps. 
The structure of a conventional general purpose computer will appear from the 
5 description below. 

In addition, the present invention also relates to a computer program product 
comprising a computer readable medium including a computer program for implementing 
the preferred methods. The computer readable medium is taken herein to include any 
transmission medium for transmitting the computer program between a source and a 

10 designation- The transmission medium may include storage devices such as magnetic or 
optical disks, memory chips, or other storage devices suitable, for interfacing with a 
general purpose computer. The transmission medium may also include a hard-wired 
medium such as exemplified in the Internet system, or wireless medium such as 
exemplified in the GSM mobile telephone system. The computer program is not intended 

15 to be limited to any particular programming language and implementation thereof It will 
be appreciated that a variety of programming languages and implementations thereof may 
be used to implement the teachings of the invention as described herein. 
1,1 Terminology 

1.1.1 Content 

20 Content is defined to be information, regardless of the storage, coding, display, 

transmission, medium, or technology. Examples of content include digital and analog 
video (such as an MPEG-4 stream or a video tape), film, music, a book printed on paper, 
and a web page. 

1.1.2 Resource 

25 A resource is a particular unit of the content being described. Examples of a 

resource include an MPEG-1 video stream, a JPEG-2000 image, and a WAVE audio file. 

1.1.3 Feature 

A feature is a distinctive part or characteristic of the resource which stands for 
something to somebody in some respect or capacity. A feature can be derived directly 
30 (ie-> extracted) from the content (eg., dominant colour of an image) or can be a relevant 
characteristic of the content. Examples of features include the name of the person who 
recorded the image* the colour of an image, the style of a video 5 the title of a movie, the 

CHP1595AU IPR34-3B_QRP149C568 I:\ELEC\CISRA\IPR\1PR34-38_GRP t\49056B.doc;PWM 



L 



-18- 



author of a book, the composer of a piece of music, pitch of an audio segment, and the 
actors in a movie, 
1.1.4 Descriptor 

A descriptor associates a representation value to a feature, where the representation 
value can have an atomic or compound type. The representation value can have an atomic 
or compound type An atomic type is defined as one of a basic set of predetermined data 
types (eg., integer, string, date, etc.). A compound type is defined to be a collection of 
one or more descriptors. The descriptor comprises a feature - representation value pair, 
where the representation value is associated with the feature. Example descriptors having 
atomic types include: 

Feature = Author; Representation Value (string) = "John Smith"; 

Feature - DateCreated; Representation Value (date) « ^1998-08-08" 
An example descriptor having a compound type is; 

Feature = Colour; Representation Value = ColourHistogramDescriptor. 
1.1-5 Description 

A description is a descriptor having a compound type pertaining to a single 
resource, 

1,1,6 Description Scheme 

A description scheme is a set of descriptor definitions !and their relationships 
(associations, equivalence, specialisations, and generalisations). The descriptor 
relationships can be used to directly express the structure of the content or t9 create 
combinations of descriptors which form a richer expression of a higher- level concept, A 
description Scheme includes within its scope a comprehensive set of description schemes, 
1,2 Descriptor Relationships j 

In order to express the information required for a description scheme, the DDF 
preferably provides a minimum set of descriptor relationships. This minimum set 
includes: 

• Generalisation/specialisation relationships, 

• Association relationships, 

• Equivalence relationships, 

• Spatial, temporal and conceptual relationships. 
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• Navigational relationships. 

The generalisation/specialisation relationships specify that a particular descriptor is a 
more specific or more general form of another descriptor and hence can be viewed by a 
5 processing application as such. For example, a cat is a type of anitnal, and hence a search 
engine searching for occurrences of animal descriptors should also select descriptions 
which contain "cat" descriptors. 

Association relationships are defined here to include descriptor containment and 
sequence and cardinality of occurrence. These relationships provide contextual 

10 information for a given descriptor and are necessary in order to provide a context in which 
a particular descriptor can be interpreted by an application. For example, a "Shot" 
descriptor which is contained within a "VideoScene^ descriptor in a video description 
scheme would be interpreted differently from a "Shot" descriptor in another context in a 
sound effects description scheme. 

15 An equivalence relationship is a form of a classification < relationship where the 

relation is not necessarily of a generalisation/specialisation; nature. Equivalence 
relationships are desirable between languages (ie., inter-languagei) and within a language 
(ie., intra- language). Typically equivalences will require the definition of synonyms 
(where two descriptors are equivalent) and quasi-synonyms (where two descriptors are 

20 equivalent to some specified extent). Also there is a need to define equivalence 
relationships between non-textual values (eg,, mean R, G and B values in an imagp) and a 
textual representative value (eg., red, green, etc.), and vice- versa. , 

Spatial, temporal and conceptual relationships between descriptors in a description 
may also be used. These relationships support the description of, neighbouring objects in 

25 an image, sequential segments in a video scene, and similar concepts in a description. 

Navigation relationships between descriptors are also desirablje. Usage of descriptions 
will often involve navigation between a component of the description and an associated 
spatio-temporal extent in the resource (such as a key frame in a video resource). 

Considered together these relationships can to some extent provide a level of semantic 

30 interoperability between different description schemes. Further levels of semantic 
interoperability could also be achieved at the application level. 
1 .3 Overview of Embodiments of Methods 
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The methods described herein are specific examples of a generalised form of a 
method for generating and processing descriptions of resources utilising a Dynamic 
Description Framework (DDF). This framework provides an object model, a platform- 
and language-neutral application programming interface (API) and a serialisation syntax 
5 for use in the description of resources, in particular audiovisual resources. The preferred 
DDF incorporates the benefits of declarative description of content with procedural 
methods for the creation and processing of descriptions and components of descriptions. 

Fig. 1A shows an overview of a method of generating a description of an 
electronically-accessible resource. In this method, a description scheme (DS) 100 A is 
10 read by a description generator 106 A which in turn generates a representation 108 A of a 
description 107 A of the resource in memory. This representation :108A is an instance of 
the Description Object Model (DesOM) of the DDF. The representation 108 A of the 
description 107 A can be serialised as an XML document 1 10A for the purposes of storage 
and transport. Preferably, both the description scheme 100A and the serialised 
15 description 1 1 OA are textual and are both readable by machines and humans. It is further 
preferable that the description scheme 100A is provided with associated procedural code, 
called DescriptorHandlerXs), so as to provide operations/processes which can 
unambiguously provide or generate descriptive information or other actions on the 
resource 104A. For example, the method in one operating mode is able to automatically 
20 generate a description 107A of the resource 104 A. In this operating mode, the processes 
of the DescriptorHandler(s) operate on the resource 104A to generate a description 107 A 
of that particular resource 104A. These description schemes 100 A and descriptions 107 A 
are defined in terms of the abovementioned DDF. 

Fig. IB shows an overview of the method of processing a description of a resource. 
25 In this method, a serialised description 100B is parsed by a processor 102B which in turn 
generates a representation 104B of the description in memory. The representation 104B is 
an instance of the DesOM of the DDF. Such a serialised description 100B may be 
generated in accordance with the method of Fig. 1A- Preferably, the processor 102B and 
description generator 106A (Fig, 1) are incorporated as one unit. The serialised 
30 description 100B refers to a description scheme 106B which may in turn refer to a number 
of DescriptionHandler(s) 108B. The serialised description 100B also refers to the 
resource 11 0B which the description describes. In this method, a set of rules may be 
applied to the DesOM representation of the description (D) 104B to generate a modified 
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DesOM representation of the description 1 12B. [The term D+ has been used to indicate 
the modified DesOM representation of the description 112B in Fig. IB.] The modified 
DesOM representation of the description 112B can be serialised as an XML document. 
This set of rules is defined by the Description Scheme 106B, The DescriptorHandler(s) 
5 108B provide further processing of the DesOM representation of the description 1 14B or 
the modified DesOM representation of the description 1 16B. In one operating mode, the 
processing method is able to compute the similarity between resources 11 OB. In this 
mode, the DescriptorHandler provides a process for computing similarity between 
descriptions of resources. The processing method is further adapted to apply a set of rules 

10 118B to the DesOM representation of the description 104B, ; The set of rules 118B 
provides one or more associated actions on the description 104B depending on the 
presence of pre-determined components of the serialised description 100B. The resultant 
output of these actions is itself a representation of a description which conforms to the 
DesOM 1 12B. Further, a description scheme may be read into memory and a set of rules 

15 provided for performing one or more associated actions on the description scheme itself 
These sets of rules are able to extend resource descriptions; translate resource 
descriptions; select one or more specific descriptions according to a queiy; visually 
present resource descriptions and many other actions. 

Fig. 1 C shows an overview of the method of encoding a description of a resource. 

20 In this method, a description scheme (DS) 104C is read by a description generator 108C 
which in turn generates a representation 1 IOC of a description of the resource injcnemory. 
This representation UOC is an instance of the DesOM of the} DDF. The description 
scheme 104C is provided with an associated procedural code, by means of a 
DescriptorHandlers (DH) ? so as to provide an encoding procedure 114C on the DesOM 

25 representation 1 10C, The encoding procedures encodes the DesOM representation 1 10C 
to provide an encoded DesOM 112C. The encoded DesOM representation 112C of the 
description can be serialised as an XML document for the purposes of storage and 
transport. The encoding procedure is preferably utilised if or compression and/or 
encryption purposes. 

30 Fig. ID shows an overview of a method of decoding an encoded description of a 

resource. This method has as its input a serialised description 104D which has been 
encoded by the method of Fig. 1C. In this method, the serialised description 104D is 
parsed by a processor HOD which in turn generates an encoded representation 11 2D of 
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the description in memory. The representation 112D is an instance of the encoded 
DesOM. The description scheme 106D provided with associatedprocedural code, called 
descriptor handlers, so as to provide the decoding operation 1 14© which can decode the 
encoded DesOM representation 11 2D so as to provide the decoded representation 116D 
5 of the description in memory, The representation 1 1 6D is an instance of the DesOM of 
the DDF. 

2. Dynamic Description Framework 
2.1 Overview 

The preferred DDF attempts to incorporate the benefits of declarative description of 
10 content with procedural methods for the creation and processing of descriptors. It 
comprises an object model, an API for the processing of descriptions, and a serialisation 
syntax. The DDF can be used to adequately describe content using these components. 

The object model provides the core semantics of the description and is based on the 
descriptor entity. This model has the advantage that the containment relationship is 
15 inherent in the model. This containment relationship is particularly important in the 
description of audiovisual resources for two reasons. First, the structure of many 
audiovisual resources has an inherent hierarchical structure (eg., a video clip contains 
shots which contain key frames, etc.). Second, the representation values for many 
descriptors can be complex datatypes that can be represented in a hierarchical fashion 
20 (eg. 7 a histogram contains bins which contain frequencies). The object model of the 
preferred DDF is called the Description Object Model (DesOM). It is discussed in 
Section 2.2. 

The preferred DDF also uses an API for the processing of descriptions. This 
enables applications and tools to perform further processing (eg., transformations, 
25 presentations, etc) on serialised descriptions. The preferred API, which is described 
further in Section 2.3, is based on the Document Object Model: called the DOM, which 
has been standardised by the W3C for use with XML documents.' 

The DesOM API also enables the application of rule-based processing, which can 
be used to: 

30 • Extend a description by inferring the presence of additional descriptors based on the 
existence or absence of stored descriptors; 
• Influence/control the presentation of a description; 
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• Select descriptions or components of descriptions; 

• Translate a stored description into another language on the baisis of requirement; 

• Transform a description to use a new description scheme. 

This rule-based processing is described in more detail in Sections 7 to 11. 
5 The tree-based structure of the DesOM (and for that matter, the DOM) is an 

appropriate representation of hierarchically structured data such as the preferred data 
model- 

The DDF preferably uses a serialisation syntax for the purposes of storage and 
transport of descriptions and description schemes. Serialised descriptions can be parsed 

10 into an instance of the DesOM. In addition, the serialisation syntax provides a means for 
expressing the descriptor relationships detailed in Section 1.2. The syntax of XML 
Document Type Definitions (DTDs) is used to express description schemes and XML 
documents to serialise individual descriptions. The expression syntax of both description 
schemes and individual descriptions is referred to as the serialisation syntax. 

15 XML is used as the serialisation syntax because of its inherent ability to express the 

containment relationship and its increasing acceptance as a form for the transmission of 
structured electronic data, A description scheme can be represented using the grammar 
of an XML DTD in which the individual element definitions represent the definitions of 
the descriptors and their relationships in the description scheme; Individual descriptions 

20 can be serialised as XML documents that conform to the DTE> containing the relevant 
description scheme. Section 2.4 describes how the preferred object model* .and the 
required descriptor relationships are expressed using the serialisation syntax. 

The use of XML as the serialisation syntax enables the possibility of DDF 
conformant descriptions to be interpreted, in theory, at two levels. First, any serialised 

25 description is able to be interpreted at an XML syntactical level. At this level the 
description could be parsed into an object model such as the DOM and a search/filter 
engine with no knowledge of the DDF could interpret the description in terms of its 
textual content (ie , the semantics of the DDF's object model are not used for the 
description's interpretation). Alternatively, the description could be parsed at a more 

30 semantic level by using the DDF object model, the DesOM, rather than the DOM, 

In practice, however, it is necessary to parse the description scheme expressed using 
the XML DTD syntax into an XML DTD where descriptor specialisation/generalisation 
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relationships are validated and explicitly realised (see Section 2.4.1-1 for further details). 
This step is necessary because no level of subclassing or inheritance is provided for in 
Version 1,0 of XML. We refer to this step as DDF interpretation and the process 
performing the step is a DDF Interpreter. To differentiate betwebn the DTD containing 
5 the DDF definition of a description scheme and the DTD to which the description (ie,, 
XML Version 1.0 document) conforms, we name the DDF DTD using an extension "ddf ' 
rather than "dtd" as is typically used for an XML DTD. 

A serialised description can then be parsed and represented using the DOM from its 
conformant DTD (ie., the DTD stored using the extension "dt4") by a standard XML 

10 Processor. This processor needs no knowledge of DDF aijid the content of the 
descriptions can be accessed at a textual level. [Textual access to the description could 
also be achieved by simply scanning the description (XML document) or using XML 
Processors that are not based on the DOM (eg,, SAX)] Alternatively, a DDF Processor 
can parse the serialised description and represent it using the DesOM from the DTD 

15 containing the description scheme expressed using DDF (ie., the DTD stored using the 
extension "ddf')* The first step of the latter process is the one of DDF interpretation. 

This process of two level interpretation is depicted in Figs. 2A and 2B, which show 
how different semantic levels of access can be obtained from a (DDF) description 
serialised using the XML syntax. The DesOm and DOM are similar in that both are tree- 

20 based structures. However, the DesOM differs from the DOM in that DesOM contains 
element nodes which have a richer interface than the corresponding element nod§s in the 
DOM. In addition, the element nodes of the DesOM can have an associated 
DescriptorHandler (H) which provides procedures that are relevant to the element. 
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2.2 Object Model 
2.2,1 Overview 

The object model adopted for the preferred DDF is based on the definition of a core 
Descriptor object. As defined in Section 1.1.4 , a descriptor can be viewed as an "feature- 
5 representative value" pair. The representative value can be of atomic type (eg. s integer^ 
string, date, etc.) or compound type, where a compound type is a collection of one or 
more descriptors. The object model is represented by the UML plass diagram in Fig 3. 
[Note that the use of capitals in Descriptor and Description impliejs the objects as defined 
in Figure 3 rather than the general terms defined in Section 1.1.4.] 

10 A Description object is defined as a specialisation of a Descriptor in which all the 

contained Descriptors pertain to a single resource. Description schemes will contain 
definitions of descriptors and descriptions which are specialisations of the core Descriptor 
and Description objects, respectively. 

In the preferred object model descriptors can represent properties and relationships 

15 of their parent descriptors. For example, a Region Descriptor for a Region Adjacency 
Graph of an image could contain a Label Descriptor (containing a textual representative 
value) and a Neighbours Descriptor (containing a representative value comprising a list of 
references to other Region Descriptors). In this example, the Label Descriptor can be 
viewed as representing a property of a region and the Neighbours Descriptor as 

20 representing a spatial relationship involving the region. Descriptors representing 
relationships (eg., spatial, temporal, conceptual) typically have representative values that 
comprise one or more references to other descriptors in the description. In Section 
2.4.1.4, a standard set of descriptors are proposed to express spatial, temporal and 
conceptual relationships. 

25 2.2.2 Descriptor Class 

Each Descriptor has an associated id, language code and| dataType enumeration. 
The id attribute provides each Descriptor with a unique identity. This identity can be used 
to reference other Descriptor objects in a description The language code attribute 
specifies the language of any text in the Descriptor's representative value. The dataType 

30 enumeration provides the data type of the representative value if that value is atomic (ie. ; 
not composed of other descriptors; see Section 2.2.3). Each Descriptor object can also be 
associated with a Descriptor Handler which provides procedural methods associated with 
the Descriptor (see Section 2.2.4). 
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Implementations of the preferred DDF object model cari implement to various 
extent the descriptor relationships detailed in Section 1.2, This approach means that 
different implementations can utilise the properties of the particular serialisation syntaxes 
adopted. Section 2.4.1 describes in detail how the descriptor relationships detailed in 
5 Section 1.5 are realised using an XML serialisation syntax. 
2.23 Atomic Descriptor Value Class 

A Descriptor's representative value can be atomic or compound (ie., composed of 
other Descriptor objects). If it is atomic, then the value is stored in an Atomic Descriptor 
Value object as a string object The data type of this atomic value is interpreted using the 
10 dataType attribute of the parent Descriptor object. Therefore the extent to which data 
typing is provided depends on the dataType attribute for partictdar implementations of 
this data modeL For example, refer to Section ^2,4.2 for data typing implementation 
details using the preferred XML serialisation syntax. 

The Atomic Descriptor Value could also be represented by a data attribute of the 
15 Descriptor class. The Atomic Descriptor Value is represented here as a class because of 
the one-to-one correspondence of this entity to a Text node in the DOM (and 
AtomicDescriptorValue node in the DesOM; see Section 4. 1 .3), 
2,2,4 Descriptor Handler Class 

In the preferred DDF, a Descriptor Handler is a class which provides procedural 
20 methods that apply to the Descriptor. The methods of the Descriptor Handler preferably 
satisfy a specified interface. The Descriptor Handler classes can; provide methods for the 
creation of a Descriptors representative value (or content) and the computation of the 
similarity between two descriptors of the same type (ie., that use the same Descriptor 
definition and hence Descriptor Handler), There is no reason why this set of procedures 
25 could not be extended if required. Fig. 3 details some examples Of the Descriptor Handler 
methods provided in the preferred implementation of the DDF. 

The methods mentioned above are preferably implemented 1 as static (class) methods 
that satisfy a specified interface (eg., see Section 4.1.2). The role of the Descriptor 
Handler is to provide unambiguous procedures for the generation and processing of 
30 Descriptors. The ability to pass parameters to Descriptor Handler methods is discussed in 
Section 3.L2.1 with respect to the use of XML as a serialisation syntax. 

Preferably, the programmatic interface for a Descriptor Handler is fixed. In other 
embodiments, the interface could be specified as an attribute of the Descriptor class or 
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specified for the description scheme. These alternative embodiments enable the 
Descriptor Handler interface to be customised for particular description schemes. 

Descriptor Handler methods can also be provided for the encoding and decoding of 
a Descriptor's representative value. Encoding methods could be provided in order to 
5 either compress (ie., reduce in size) the serialised description and therefore more 
efficiently store and transport the description, or alternatively to encrypt the Descriptor's 
representative value. 

In the case of encoding for compression, the encoding methdd could vary depending 
on the type of data to be encoded. For example, a Descriptor with a textual representative 

10 value could use a text compression method (eg., LZW), whereas a Descriptor that 
represented a colour histogram structure of an image resource may encode the bin 
frequencies of the histogram using a form of entropy encoding (ie., most commonly 
occurring frequencies are represented by codewords requiring fewer bits). Encoding for 
encryption could be used to allow only privileged users access to die Descriptor. Standard 

15 encryption methods (eg., public key encryption) could be used. 
2-2.5 Description Class 

The Description has some additional attributes to those of the Descriptor. It has an 
associated resource which contains either the URI or ENTITY- of the item of content 
being described. It also contains a reference to the data when that resource was last 

20 modified and an attribute that contains the URIs or ENTITIES onsets of rules that can be 
applied to the Description. Rule-based processing of descriptions is discussed further in 
Section 7. 

Since a Description object is defined as a specialisation of the Descriptor object. 
Description objects can be treated as Descriptor objects in other descriptions (ie., the 
25 attributes of the Description are ignored). In an alternate dat^ model, the Description 
object can contain both Descriptor and Description objects, j With this data model 
Description objects can exist in another tree of Descriptors and refer to resources other 
than that of the root description. 

Another alternative implementation could use a data model which did not include a 
30 Description object, since a Description is essentially the same as a Descriptor having a 
compound representative type. In this case the additional attributes of the Description (ie 
resource, dateResourceLastModified and ruleSets) would be treated as attributes of the 

j 
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Descriptor. With this data model the resource would only need to be specified at the top 
of the Descriptor tree where it was relevant. 
2.3 API for Processing of Descriptions 

The inherent containment property of the core Descriptor object is represented by a tree- 
5 based processing model (ie. ? parent-children data model) where each node of the tree is 
either a Descriptor or Atomic Descriptor Value object, [Atomic Descriptor Value objects 
can only exist as leaf nodes of the tree.] The DesOM also contains references and 
navigational links between nodes in the tree. References are typically used to indicate 
relationships (eg., spatial, temporal and/or conceptual) between Descriptor objects. 

10 Navigational links are used to provide browsing properties for the description and enable 
linking between Descriptor objects in the description and spatio-temporal extents in the 
resource (eg, 3 a particular frame in the video stream being ddscribed). A schematic 
depicting the description processing model is shown in Fig. 4. 

For a description to conform to the preferred DDF, the root Of the DesOM must be a 

15 Description object. In other words, the root must specify the resource to which the 
description refers. Since a Description object is just a specialisation of the Descriptor 
object, any Description object can become a sub-tree of another^ Description object. In 
other words, a new Description object can be created from a s^t of related Description 
objects. This process is shown hi Fig. 5. 1 

20 The DesOM extends the DOM by providing the required 

generalisation/specialisation relationships for descriptors, data typing for, atomic 
representative values for descriptors, DescriptorHandlers (H) and reference and 
navigational links. The DOM provides a standard set of objects for representing XML 

documents, a standard model of how these objects can be combined, and a standard 

i 

25 platform- and language-neutral interface for accessing and manipulating them. The DOM 
representation of an XML document is a tree structure where the; content of an element is 
represented as child nodes of the element. The DOM specifies; interfaces which can be 
used to manage XML documents. In other words, it can be implemented in any (or nearly 
all common) programming languages. 

30 Similarly only interfaces are specified for the DesOM. These interfaces can be used 

to process XML documents that are DDF conformant. Just as an XML (DOM) Processor 
must implement a DOM interface, a DDF Processor must implement a DesOM interface 
(see Fig. 2). As mentioned in Section 2.1, a DDF Processor first performs an 
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interpretation step in which the generalisation/specialisation relationships of descriptors is 
validated and processed in a Version 1.0 XML DTD form, [Invalid subclassing in the 
description scheme expressed using the DDF and the syntax of XML DTDs should result 
in a description scheme parsing error.] The DDF Processor can then either parse the 
5 description into a DOM and transform that structure into a J DesOM or parse the 
description directly into a DesOM. 

Essentially the DesOM differs from the DOM in that element and text nodes are 
replaced by the richer interfaces of Descriptor and Atomic Descriptor Value nodes. 
Interfaces for these nodes are described in Section 4 and section 6-3- A basic DesOM 

10 implementation could provide just that interface, however a more expansive 
implementation might provide some level of interpretation of the reference and 
navigational relationships. For example, a set of spatial, temporal and conceptual 
relationships could be defined for the DDF (see Section 3.1.3) and these could be 
interpreted at the DesOM level. 

15 Implementations of the DesOM could optionally execute Descriptor Handler 

methods to create, encode or process descriptors. For example, a DesOM implementation 
might implement a Descriptor Handler's method to create the content for a Descriptor if 
the content did not already exist. 

The DesOM provides a basis for the further processing of descriptions. The tree- 

20 structure of the DesOM makes it amenable to rule-based processing where rules consist of 
a pattern and an associated action, Such processing could be performed by tools which 
implement the DesOM interface to process DDF descriptions. Rule-based processing is 
discussed further in Section 7 to 1 1, 

25 i 

2.4 Serialisation Syntax 
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The serialisation syntax preferably used for the storage and transport of descriptions 
and description schemes is XML Version 1.0. The XML standard was developed as a 
subset of Standard Generalised Markup Language (SGML). An XfvlL document contains 
one or more elements, the boundaries of which are either delimited by start and end tags, 
5 or by an empty-element tag. Each element is identified by its name, sometimes also 
called its "generic identifier" (GI) and may have a set of attribute specifications. Each 
attribute specification has a name and a value. For further details on the XML Version 
1.0 standard, reference is made to the W3C website HTTP://www. w3.org/TR/199S/REC- 
xml-19980210. 

10 The preferred DDF uses a set of core elements which can be defined in an DDF 

Core DTD. A SGML-like DTD syntax is used to define element types and their 
associated attributes (as specified in the Version 1.0 XML standard). Each description 
can be represented by an XML document. This document (ie. ? the description) refers to 
the DTD (ie., the description scheme) to which the description conforms. In other words 

15 the description is of the type specified by the DTD (see Fig. 6) 

The DDF Core DTD needs to provide definitions for the core elements required for 
the expression of the object model. The element definition that is central to the DDF is 
that of the Descriptor element. All descriptors can be defined as subclasses 
(specialisations) of this core element. For example, although a Description is defined to 

20 be a collection of descriptors pertaining to a single resource it is defined as a subclass of 
the Descriptor element. Other subclasses of the Descriptor element are used to /provide 
linking functionality between the descriptors and the resources; being described (see 
Section 3.1.4). 

The data modelling requirements of the DDF are more extensive than those provided 
25 by the XML Specification version 1.0. Specifically the serialisation syntax of the DDF is 
able to: 

• Express the required descriptor relationships (see Section 2); 

• Provide data typing for the (atomic) representative value of a descriptor; 

These requirements are addressed in Sections 2.4.1 and 2.4.2 with respect to using 
30 Version 1.0 of the XML standard as the serialisation syntax. 
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2-4.1 Expression of Descriptor Relationships 
2,4.1*1 Generalisation/Specialisation Relationships 

Version 1.0 of the XML specification does not provide for the specification of 
generalisation/specialisation relationships. In addition, subclassing and inheritance in 
5 marked up documents is not well-defined. An element type is a subclass (specialisation) 
of another element type, the superclass, if it is substitutable wherever the superclass 
element occurs and is defined to be a subclass of the superclass. It is not essential for an 
element to be defined as a subclass of another element. The superclass can be viewed as a 
generalisation of the subclass. The notion of inheritance can be viewed as a code-saving 

10 mechanism which allows one element type to get (inherit) the: properties of another 
element type "for free". 

The preferred subclassing/inheritance guidelines for single subclassing/inheritance 
is described below in Sections 2.4.1.1.1 to 2.4. LI. 3. Multiple inheritance can be 
extended from the single subclassing/inheritance. 

15 2.4.1.1.1 Content Model Inheritance 

A subclass should faithfully implement a base class's interface. Therefore, if a base 
class has a content model of "ANY" then a subclass can have eitjher an "ANY" content 
model or a more restricted content model. This is necessary for the subclass to be 
substitutable for the parent class. This is a somewhat different scenario from object 

20 oriented programming (OOP) where a subclass must accept any input that its super 
(parent) class can. The content model of an element should be viewed as "output" not 
"input". If each element is considered as an object having methods to retrieve its content, 
then a subclass must also be able to satisfy these methods. Each element type in a content 
model can be viewed as having a role and the roles of a subclass's content model must 

25 match up with those of its parent class. A subclass cannot make more flexible or extend 
components of the content model of its parent class, however it can implement new child 
elements that will be ignored when that element is treated as its parent class. 

For example, if AA S BB and CC are subclasses of A, B and C ? respectively and A 
has a content model of (B a C) then the following are all valid content models of AA; (BB ? 

30 CC), (B, C), (BB, C) and (B, CC). The content models (B, C, D), (BB, CC, D) and (D, B, 
C) are also valid content models for AA because they match "roles" for (B), (C) and (B, 
C). In addition element, AA can contain child element D which will not be visible if 
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element AA is to be treated as an instance of element A. The content models (B) and (C) 
are invalid because of the "role" of (B, C) in the content model of A is not matched. 

It would be possible to allow the content model for a subclass to be left unspecified 
in which event the subclass's content model would default to be that of the superclass. 
5 Preferably, unspecified content models should not be allowed as 'they do not represent a 
valid construct using XML/SGML DTD syntax. 
2.4.1.1.2 Attribute Inheritance 

The same subclass and inheritance notions apply to attributes, however attributes 
are more intrinsically amenable to concepts of subclassing than content because they are 
10 "random access" in some sense as are methods in OOP. A subclass can declare new 
attributes which are essentially ignored when the subclass is treated as it parent class. 
However, a subclass cannot extend, or make more flexible, attributes of the parent class. 

The attribute defaults are only considered when assessing whether an attribute 
definition has or has not extended that of its parent class. Consequently a subclass and its 
15 specified superclass should have the same attribute type, and only [the attribute default can 
be further restricted in the subclass, Valid restrictions of attribute default definition are as 
in Table 1. In addition, if the superclass has a default declaration of M #FIXED" and the 
value of the default can be interpreted as an element name then preferably the value of the 
default can be further restricted to a be a subclass of that element name. 



Table 1. Peimitted restrictions of the attribute default declaration in a subclass. 



Superclass Attribute 
Default Declaration 


Subclass Attribi 
#IMPLIED 


ite Default Declai 
#RBQUIRED 


cation 

i 

"value" 

i 


#FIXED "value" 


#IMPLIED 


V 


V 


v . 


V 


#REQUTRED 




V 


v ; 


V 


"value" 










#FIXED "value" 






i 





2.4.1,1,3 Implementation Details 

In order to implement this subclassing/inheritance model using Version 1 ,0 of the 
XML Specification and the DOM, the superclass (or superElement) for an element is 
specified as an attribute in the element's defined attribute list. It is believed that this is 
not ideal and that subclassing information should be part of the element's definition. For 
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example, the keyword c TYPEOF" has been suggested as a means of representing 
subclassing information (ie., <! ELEMENT Cat TYPEOF Animal>)L 

The subclassing/inheritance implied by the use of the superEleinent attribute needs to 
be interpreted and validated against the provided guidelines for subclassing/inheritance. 
Failure to conform to these guidelines should result in a description scheme parsing error. 
Also, in order for a serialised DDF description to be a validl XML document, the 
description needs to conform to a valid XML DTD. Therefore the DDF description 
scheme that is expressed using the syntax of XML DTDs needs to be parsed to create an 
XML DTD in which all the inheritance aspects of the subclassing relationships are 
processed. This involves: 

• Making explicit content models which depend on subclassing (this may involve 

extending content models so that they represent valid XML DTD content models in 

i 

the absence of subclassing semantics); 

• The addition of inherited attribute definitions to subclassed Descriptor definitions. 
2.4.1.2 Equivalence Relationships 

The location of described resources can be achieved by thejmethod by formulating 
requests directly based on a description scheme or by more unstructured queries in which 
the contents of a description scheme are unknown. Typically tt^e former approach will 
result in a more satisfactory result because the query is specifically formulated for the 
form of the descriptions. However, in some cases a query might be formulated without a 
(complete) knowledge of a description scheme (and hence use different terms than those 
used in the description scheme) or in a language other than that used by particular 
descriptions. 

As highlighted in Section 1 .2 there are three types of equivalences^ 

• Intra-language equivalences (ie., synonyms or quasi-synonyms); 

! 

• Inter-language equivalences (ie., translations); 

• Inferred equivalences between textual and non-textual representative values. 
Known intra-language equivalences could be incorporated into a descriptor's definition 
using an alias or sameAs attribute for elements. However, applications and tools that 
provide a level of intra-language equivalence interpretation exist and therefore it was 
considered unnecessary to provide this functionality. Separate search/query/filter engines 
can ultimately provide some level of intra-language equivalence interpretation. 
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It is desirable to provide a means for inter-language equivalence as queries will not 
always be formulated in the same language as the description. Although seme degree of 
redundancy can be tolerated in a description scheme (ie.> disscriptors in different 
languages could be defined), it is not generally acceptable to express a description in 
5 multiple languages. The method can translate a parsed description into the language of 
the query by processing a set of rules that is defined for the description scheme. This set 
of rules effectively replaces the descriptors in the DesOM with equivalent descriptors in 
the same language as the query- This method provides a controlled mapping between 
descriptors in different languages rather than allowing a mappinjg to be estimated by a 
10 translation ability in the search/query/filter engine. 

Equivalences between non-textual and textual descriptors j can be provided in a 

similar manner. For example, if the colour of an object in an image is stored as a (R, G, B) 

i 

value then a rule could instantiate another descriptor in the t)esOM that maps the 

i 

particular (R, G, B) values to particular colours expressed as a texjt string (eg., red, green, 
15 blue, orange, etc)* 

The rules are stored as a rule set that can be specified as par^ of the description. The 

extra or translated descriptors are not serialised and are only generated when they are 

i 

needed. In other words, they only exist in the DesOM and not in the XML document that 
represents the description. Rule sets are a way of providing a richer, more flexible, 
20 description at the time of the description being processed without [increasing the overhead 
of storing redundant information. 
2.4.1.3 Association Relationships 

i 

Association relationships specify the context in which a; defined Descriptor can 
occur. The context includes relationships such as containment j[eg. s Descriptor A must 
25 occur within a Descriptor B), sequence (eg., Descriptors A, B, ajid C must occur in that 
order), and cardinality (eg. 5 Descriptor B can occur only once in an instance of Descriptor 

a). ; 

To a large extent these association relationships can be specified in an XML DTD 
using an element's content model. A content model is a simple grammar governing the 
30 allowed types of child elements (ie., containment) and the order ift which they are allowed 
to appear. Group connectors [and (comma), or (vertical bar)] are used to specify the order 
in which child elements can appear within the element. Occurrence indicators [one or 

i 

more (+), 2ero or more (*) s or zero or one (?)] are used to specify the cardinality or 
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oocuiTence of the child elements in the element's content. Element content models are 
described in Section 3,2.1 of 71. The XML content model 1.0 does not allow a specific 
non-zero cardinality to be defined (eg., an image can contain! 0 to 20 objects) and 
consequently this association property is not provided in; the preferred DDF 
5 implementation. 

2.4.1.4 Spatial, Temporal and Conceptual Relationships , 

Many descriptors will need to be able to model spatial, tepporal and conceptual 
relationships often in addition to association relationships. For example, a Region 
Adjacency Graph which describes an image, comprises a graph object that contains a set 
10 of regions. In addition to being part of the graph object, each rjegion also has a set of 
neighbouring regions (ie., spatial relationships). These relationships can be described 
using references to the relevant descriptors in the description. 

I 

In the method, these relationships are represented as Descriptors having atomic 
Descriptor values with IDREF or IDREFS data types. A set of core relationship 
15 descriptors is defined in the DDF Core DTD to enable DesOM implementations to realise 
a greater extent of semantic interpretation. Examples of the types; of descriptor definition 
to are included are provided in Section 3.1.3. 

2.4.1.5 Navigational Relationships ; 

Many applications may require that descriptors can be explicitly linked to spatially 
20 and/or temporally localised extents in a resource. Although the resource is typically that 

i 

being described, this is not always the case. The links should ienable navigation from 
descriptors to indicated locations in a resource (eg., from a descriptor to a spatially and 
temporally localised extent in a digital video stream)* 

The means for expressing these links has been derived from] an existing approach to 

25 this problem, namely the HyTime standard, which uses locaticjn address elements, or 
locators. This method requires that the resource must be declared as 311 external entity in 
the description. Link elements are then declared to create contjextual (having a single 
linkend) and independent (having more than one linkend) links between locations in the 
description and extents in the declared entity. Locators provide a means for addressing 

30 extents in the resource being described. 

The Locator and Extent elements defined in the DDF Corel DTD are much simpler 
than those specified in the HyTime standard as the latter provided] more than was required 
for the DDF requirement of linking. Also, because it is difficult to envisage all the 
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possible different forms of locators required for the different medik types it was believed 

l 

that description scheme designers should not be limited in the scope of their design of 
required locators. 

2.4.2 Expression of Specific Data Types 

5 The content model for an element can specify the order and cardinality of allowed 

child elements (see Section 2.4.1 ,3)> that the element has EMPTY! or no content, that the 
element has parsed character data (ie., #PCDATA), or some mixture of parsed character 

data and child elements (ie., ANY). [The allowed content models \>f elements are detailed 

1 

m Section 3.2.1 of XML 1.0 WC3 recommendation]. If the conteht of an element is used 
10 to store the representation value of a feature (eg., "DateCreated"),ithen the content model 
of the relevant Descriptor would need to be "#PCDATA" (or "ANY") and the content 
would be represented as a character string. Although this might be acceptable for a 
textual interpretation of the description, this form of representation does not permit more 
advanced queries where, for example, descriptions may be requirjed to be selected if the 
15 "DateCreated" feature has a representation value that is later than isome provided date. In 
other words, it is necessary to know how to parse the character content of the Descriptor 
(ie., the Atomic Descriptor Value). 

The serialisation syntax of the DDF provides data typing of an element's content by 

j 

using a dataType attribute for the element. Although it woul£ not be explicit for a 

20 Version 1.0 XML (DOM) Processor, a DDF Processor can use tijie data type attribute to 

interpret an element's content appropriately. Datatyping of elejment content has been 

considered as part of the XML working group discussions ajnd hence it would be 

preferable if the DDF could remain consistent with the XML standard. 

In addition to the basic data types (eg., integer, floating-pjoint value, string, date, 

25 time, etc.), the dataType attribute should allow types such as ID, jlDREF and ENTITY in 

order to enable Atomic Descriptor Values to represent references jto other Descriptors and 

links to entities external to the description. The XML concept of ENTITIES is preferred 

to using a URI data type in that the ENTITY type allows a URI to be linked to a type of 

the entity (eg. 7 JPEG image, Java class, etc.), 

30 2.5 Implementation Issues 

An implemented DDF Processor could use publicly available software (eg. IBM's 

XML parser; to parse descriptions into a DOM structure and theiji the method transforms 

i 
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this structure into a DesOM structure. The Java language is preferably used to implement 
Descriptor Handler classes because of its cross-platform properties. 

Actual implementations of a DDF Processor would not need to create a DOM as an 
intermediate step and could parse the XML document directly into a DesOM structure 
5 using the DDF description scheme. Such a processor would njed to first interpret the 
subclassing information in the DDF description scheme (see Fig. ^A). 

A DDF Processor implementation could also take advantage of other core 
relationship descriptors (see Section 2 AAA) to provide a richer semantic interpretation of 
descriptions. Implementations could also interpret the linking elejments when providing a 
10 graphical representation of descriptions and incorporate rule-ejagines to process rules 
which might be applied to the DesOM. j 

3. Serialisation Syntax Specification I 

i 

3.1.1 Element Definitions 

i 

The preferred DDF includes the definition of a set of clore elements using the 

15 XML/SGML DTD syntax. This set, is preferably stored in a core, or set of core, DTDs, 

I 

Appendix A contains an example of such a DTD, Core.ddf. i Note that we use the 

extension ,r ddf 1 to differentiate this document from an XML DTP which would typically 

i 

have the extension "dtd". A DDF set of definitions needs to have its 
subclassing/inheritance properties (eg., attribute inheritance \ from super elements) 
20 processed before a description can be interpreted with respect to sfet of DDF definitions. 

The set of core elements can be used as a basis for the definition of application X>TDs or 
description schemes. The element definitions in the Core.ddf effectively provide a set of 

"foundation" elements from which description schemes can be bajsed. 

i 

This specification of the core element definitions for the proposed DDF is based on 

i 

25 Version 1.0 of the XML Specification. Elements that are iijcluded in die proposed 
Core.ddf are named according to the naming conventions used! for Java classes (ie., all 
words in the name are capitalised and concatenated). \ 
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3.1,2 Core DDF Element Definitions 
3,1.2-1 Descriptor Definition 

i 

The Descriptor element is the basic element which provides the data modelling 

I 

properties detailed in Section 2.2, Any element definition rjequiring any of these 
5 properties should be represented as a subclass of this element. The element is the markup 
equivalent of the object class of an object-oriented programming language. 

The content model for the Descriptor element needs to allow for either parsed 
character data (atomic representation value) or one or more Descriptor elements 
(compound representative value). The content model of the Descriptor element is defined 

10 to be "ANY" so as to allow the necessary content and be a validj XML DTD syntactical 
construct. However in order to control content models more tightly, it is also possible to 
define two subclasses of the Descriptor, the Atomic Descriptor and the Compound 
Descriptor. The content models of these subclasses could then Have the more restricted 
content models of #PCDATA and (Descriptors), respectively. | 

15 In specialisations of the basic Compound Descriptor element, the "Descriptor*-" 

would need to be interpreted by the DDF Interpreter as one |or more Descriptor or 
subclasses of Descriptor elements. Specialisations of the Descriptor element that use this 
content model by default may have their content model extended to "ANY" during the 
DDF interpretation process (see Section 2.4. L 1.3) in order to form a valid XML DTD for 

20 a description scheme- 



<! ENTITY % DataTypes 


"(Int | Float I Double 


String | Dafe | Time | ED |-IDREF | 


IDREFS | ENTITY | ENTITIES)"> 




<! ELEMENT Descriptor (ANY)> 




<!ATTLTST Descriptor 






id 


ID 


#IMPLIED 


xmlilang 


CDATA 


"en" 

1 


dataType 


%DataTypes; 


"ptring" 


superElement 


NMTOKEN 


Implied 


handler 


ENTITY 


#implied 


> 
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Attribute id \ 

The value of this attribute provides a natural way to refer (to a particular element 
(eg., in references). It must be unique for the document. ! 
Attribute xmhlang ! 
5 The attribute xmhlang is included in Version 1.0 of the XML Specification. It 

specifies the natural language or formal language in which the content (of the element) is 
written. The default language used by the Descriptor element is English. If a description 

scheme was defined in French, for example, then one approach would be to define a 

j 

FrenchDescriptor in which the value of xmhlang was FIXED to "fir", and then derive all 
10 application descriptors from the FrenchDescriptor element ' 
Attribute dataType 

Preferably, the definitions of many descriptors require some control over the data 
type of an element's character data content. ; 

The allowed data types for character data content are specified by the (XML) 

15 internal parameter entity, DataTypes (see above)* The dataType attribute is only utilised 
if the content model for the Descriptor contains #PCD ATA and the provided content for 
the Descriptor contained character data, hi other words, if the content of a Descriptor is 
specified to contain child elements (ie., a compound representative value) then the 
dataType attribute is not used. In an alternative implementation, the allowable data types 

20 could include a "Compound" type which would make the use of] a compound descriptor 
more explicit in its definition. j 

Character data content of a Descriptor is represented by a |DDF Processor using a 
AtomicDescriptorValue node (see Section 4 1.3 for the interface specification) rather- than 
a Text node as used by a DOM Processor. 

25 The default value of the attribute dataType is "String".| This means that the 

dataType attribute does not need to be included in a Descriptor element's definition if the 
content of the element is to be treated as a string. Preferably, tiie DDF Processor dates 
and times are based on the profile of ISO 8601. The types, 
ENTITY/ENT1TIES/ID/IDREF should be parsed as defined for the XML Version 1 

30 standard. 

Although the data type of the Descriptor element's character data content cannot be 
directly used by XML version 1.0 and DOM version 1.0 specifications, it might in some 
way assist textual access to the description. Also placing the data type of the character 
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data content in an attribute is consistent with many current proposals for data typing in 
" XML. : 

Some Descriptors will require their representative value to be limited to a list of 

! 

possible values (ie. 5 an enumeration). In these cases, it is preferable to construct 
5 Descriptor elements (having an EMPTY content model) for eaph of the enumerated 
values and then specify the enumeration in the content model fori the parent Descriptor. 
An alternate approach is to include an enumeration data type and use a #PCD ATA content 
model . 

Attribute superElement \ 

i 

10 The value of this attribute is an element name which is the parent (or super) element 

of the Descriptor element. The parent element's definition must bej available. Subclassing 
is implemented as described in Section 2.4. 1.1. j 

The information in this attribute is used by the DDF interpretation process (see Fig, 
2) to validate the defined subclassing and to process the inheritance of attributes. When 

15 accessed at the DOM level 3 this attribute provides only descriptive information about the 
immediate generalisation of the element. When processed af: a DesOM level the 
subclassing relationship (s) for the element are represented as a i^ode list or inheritance 
tree (see Section 4. 1 . 1). < 
Attribute handler \ 

20 The value of this attribute specifies an external entity for a ijescriptor Handler to be 

used to provide methods for the Descriptor element. The Descriptor Handler is, a class 
which contains methods that conform to a specified Descriptor: Handler interface (see 
Section 4.1.2). 1 

The Descriptor Handler is specified using an ENTITY whic|i can be defined in the 

25 description scheme (preferably before the elements of the scheme are defined). The 
ENTITY declaration can use a NOTATION to declare the type of the external entity and a 
helper application required to process the ENTITY. La the example below, a NOTATION 
is declared for a JavaClass type and this type is linked to the "Java" helper application (ie., 
Java virtual machine). An individual Java class in then declared using an ENTITY 

30 declaration which uses the JavaClass NOTATION, 
<!NOTATION JavaClass SYSTEM "Java"> 
<! ENTITY MyDescHandter SYSTEM "MyDescHandler.class' 1 NDATA JavaClass> 
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Preferably, it is assumed that the methods provided by the Descriptor Handler do 
not require any parameters that are not available from the DesOM (eg., resource from a 
Description element). If methods of a Descriptor Handler require parameters to be set 
from individual descriptions, then attributes of a specialisation ofjthe Descriptor element 
5 can be used to hold the parameter values. A Descriptor Handler CQuld then have a method 
to set the parameters from the attribute values in the DesOM. 
3.1.2.2 Description Definition 

A Description element is defined as a subclass of the ^Descriptor element. It 
represents the root node of an instance of the DesOM and should be the root element of a 
10 serialised description (ie,, an XML document). j 

The content model for the element is defined as one or more Descriptors. This is a 
restriction of the content model of the Descriptor element. As with the Descriptor 
element, definitions of specialisations of this element need to be interpreted by the DDF 
Interpreter as one or more Descriptor or Descriptor subclass elements. 



15 


<!ELEMENT Description (Descriptor*-)> 






<!ATTLIST Description 






superElement NMTOKEN 


##IXED "Descriptor" 




resource ENTITY 


#REQUIRED 




dateResourceLastModified CD ATA 


#1MPLIED 


20 


ruleSets ENTITIES 

> 


#!MPLffiD 

i 



Attribute superElement 

Although the attribute superElement is inherited from the Descriptor element's 
25 definition, it is redefined here to declare that the Description eleitient is a subclass of the 
Descriptor element. The default superElement is declared as #FIXED so that instances of 
the Description element cannot redefine the superElement ; value. Note, that a 
specialisation of the Description element can further restrict this default attribute value by 

i 

specifying an element name that is a subclass of the Descriptor element (see 2 Al .1.2). 
30 Attribute resource 

This value of this attribute should contain an entity which references the resource 

being described by this description. The resource must have beei> declared as an entity in 

i 
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the description before the Description can be declared. The resource type can be obtained 

i 

by using a NOTATION, defined in either the description scheme! or in the Core.ddf, to 

i 

describe the type of entity: 

eg., <!NOTATION MPEG-2 SYSTEM "MPEG-2Player":>, 
5 The NOTATION can then be used by an external ENTITY declaration in the 

DOCTYPE declaration of the description: 

eg., <!ENTTTY MyVideo SYSTEM u MyVideo.mpg" NDATA MPEG-2>. 
Note, that this method of referencing the resource beii^g described not only 
identifies it as an MPEG-2 resource but also provides the namej of a processor (helper 
10 application) for the resource type. j 
Attribute dateResourceLastModified 

The value of this attribute is a string representation of the date that the resource was 
last modified. At any stage a process can check to see if this dat^ has changed (by string 
comparison), and update the description if necessary. j 

15 Attribute ruleSets ! 

! 

The value of this attribute contains one or more external ENTITIES. Each ENTITY 
refers to an XML document that contains a set of rules that ! can be applied to the 
description (see Section 7). j 

3.1.3 Descriptors Representing Spatial, Temporal and Conceptual Relationships 

i 

20 A set of Descriptor elements have been included to provide spatial, temporal and 

conceptual relationships between descriptors. These elements are preferably a part of the 
core DDF elements rather than specified in individual application description schemes in 
order to improve the semantic interpretation of description. These relationship Descriptor 
elements can have either atomic or compound representation values. The element set 

25 below is included more by way of example rather than attempting to demonstrate a 
complete list of the types of relationships that need to be modelle4- 

i 
i 



30 

<! ELEMENT Parallel Sequence (Descriptor^ 

<! ATTLIST ParallelSequcnce , 
_ -■ — i . 
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superElement 



NMTOKEN 



#FIXEE) "Descriptor" 



<!ELEMENT SerialSequence (Descriptor+)> 
<! ATTLIST SerialSequence 

superElement NMTOKEN 

> 

<!ELEMENT Neighbours (#PCDATA)> 

<! ATTLIST Neighbours 

superElement NMTOKEN 
dataType %DataTypes 5 

> 

<!ELEMENT Before (#PCDATA)> 

<! ATTLIST Before 

superElement NMTOKEN 
dataType 0 /cDataTypes; 

> 

<! ELEMENT After (#PCDATA)> 

<! ATTLIST After 

superElement NMTOKEN 
dataType %DataType$; 

> 

<! ELEMENT InFrontOf (#PCDATA)> 

<! ATTLIST InFrontOf 

superElement NMTOKEN 
dataType %DataTypes; 



#FIXEq "Descriptor" 



#FIXED "Descriptor" 
#FIXEfl "IDREFS" 



#FDCEI} "Descriptor" 

l 

#FEXED "IDREFS" 



#FDCEEj "Descriptor" 
#FTXEli "IDREFS" 



#FIXED "Descriptor" 
#FDCEE) "IDREFS" 
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<IELEMENT Behind (#PCDATA)> 


i 


<!ATTLIST Behind 




superElement NMTOKEN 


#FIXED ^Descriptor" 


dataType %DataTypes; 


#FD£ED !TDREFS" 


> 





i 

3-1-4 Elements Representing Navigational Relationships j 

The preferred DDF also includes some core elements that i enable the linking of 
descriptions to spatio-temporal extents of the content being described. A spatio-temporal 
extent is defined to be a section of the content that is spatially and/dr temporally localised. 
For example, a spatio-temporal extent of a digital video signal might be represented as a 
rectangular region that extends for a number of frames, A contextual link, CLink, is 
defined to represent the common cross-reference or navigational lipk, A Clink connects 
the location in the description where the link occurs to another location. In other words, a 
CLink has a single linkend attribute. An independent link, or ILink, is also defined for 
applications that require links connecting more than two locations or stored separately 
from the link's location in the description. These elements are dpfined as subclasses of 
the basic Descriptor element so that they are interpreted by the E|DF and represented as 

nodes in the DesOM, Since these elements do not require anyj of the data modelling 

i 

properties described in Section 2.2, there may be a case for allowing elements, suoh as the 
set defined below, to not be based on the Descriptor element biiit still interpreted by a 
DDF processor. ; 



The definitions of these linking elements are included in Core.ddf. Note it might be 
preferable to include the definition of the core spatio-temporalj linking elements in a 
separate (ddf) DTD. j 



<!ELEMENT CLink (#PCDATA)> 






<!ATTLIST CLink 






superElement 


NMTOKEN 


#FTXED "Descriptor" 


dataType 


%DataTypes, 


#ETXED "BDREF" 


> 

<!ELEMENT ILink (#PCDATA)> 




l 

1 . . 
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<!ATTLIST Dink 








iMM 1 OKEN 


#FIXED Descriptor ' 


dataType 


%DataTypes; 


#FIXED "IDREFS" 


> 







i 



The core Locator element simply provides an address for the! location of one or more 

i 

Extent elements within a particular resource. The value of the resource attribute 
identifies the resource using an ENTITY that has been previously declared in the 
description. This requires that the Core.ddf also includes a j sufficiently rich set of 
NOTATIONS that include the types of resources that are going to be referenced by 
entities (eg., JPEG, TIFF, MPEG-1, MPEG-2, etc.). An instance of a Locator must 
contain one or more instances of an Extent. It is desirable to specify the resource even if 
it is the same resource specified for the description. 

Several subclasses of Extent elements are defined in the Core.ddf. The definitions of 
these elements are included below. These element definitions prjovide an example of the 
types of Locator and Extent elements that could be required. ; 



<! ELEMENT Locator (Extent+)> 
<! ATTLIST Locator 


i 


superElement NMTOKEN 
resource ENTITY 

> 


#FIXED "Descriptor 

#REQUIRED - 

i 


<! ELEMENT Extent (Descriptor*-^ 
<! ATTLIST Extent 


i 
1 

i 
i 

i 


superElement NMTOKEN 

> 


#|FIXED "Descriptor" 


<! ELEMENT ImageExtent (Descriptor+)> 
<! ATTLIST ImageExtent 

superElement NMTOKEN 


#FIXED "Extent" 

i 


<!ELEMENT RectbnageExtent (RecUmageExtentXO, RectlrimgeExtentYO, 
RectltnageExtentHeight, RectImageExtentWidth)> 
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<!ATTLIST RectlmageExtent 

superElement NMTOKEN 
> 

<!BLEMENT RectlmageExtentXO (#PCDATA)> 

<! ATTLIST RectlmageExtentXO 

superElement NMTOKEN 
dataType %DataTypes; 

> 

<! ELEMENT RectlmageExtentYO (#PCDATA)> 

<! ATTLIST RectlmageExtentYO 

superElement NMTOKEN 
dataType %DataTypes; 

> 

<! ELEMENT RectlmageExtentHeight (#PCDATA)> 

<! ATTLIST RectlmageExtentHeight 

superElement NMTOKEN 
dataType %DataTypes; 



#FIXED ; "ImageExtent" 



#FPCED "Descriptor" 
#FIXED Tht" 



i 

#FIXED "Descriptor" 
#FIXED "Int" 



#FJDCED "Descriptor" 
#?ixED "Int" 



<! ELEMENT RectlmageExtentWidth (#PCDATA)> 
<! ATTLIST RectlmageExtentWidth 



superElement 
dataType 



NMTOKEN 
%DataTypes; 



#FjlXED "Descriptor" 
#ljlXED "Int" 



<! ELEMENT VideoExtent (VideoExtentStart, VideoExtentEi*d, ImageExtent?)> 
<! ATTLIST VideoExtent ; 

superElement NMTOKEN #IfIXED "Extent" 
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<• ELEMENT VideoExtentStart f#PCDATAY> 






<! ATTLIST VideoExtentStart 


- 




supplement NMTOKEN 


#FlXED "Descriptor" 


5 


dataType %DataTypes; 

> 


#FJXED "Int" 




<! ELEMENT VideoExtentEnd (#PCDATA)> 


j 




<! ATTLIST VideoExtentEnd 






superElement NMTOKEN 


#FIXED "Descriptor" 


10 


dataType %DataTypes; 

> 


#FIXED "Int" 



4. DesOM API Specification 

The DesOM interface extends the existing DOM Object Model (DOM) interface 

15 specification. The DOM is a platform and language-neutral interface that will allow 
programs and scripts to dynamically access and update the content, structure and style of 
XML and HTML documents. It provides a standard set of objects for representing HTML 
and XML documents, a standard model of how these objects cin be combined, and a 
standard interface for accessing and manipulating them. Vendors can support the DOM 

20 as an interface to their proprietary data structures and APIs, and content authors can write 
to the standard DOM interfaces rather than product-specific | APIs, thus increasing 
interoperability on the Web. i 

The DOM interface does not stipulate how its associated methods are to be 
implemented. For, example, the method getElementsByTagNampO (Appendix K) must 

25 satisfy the DOM interface, but can be implemented in any majiner as a developer so 
choses. The implementation of the methods associated with DesOM and DOM interfaces 
are not essential to the invention and will not be described further.! 

The DOM Level 1 Specification is now publicly available; it has been reviewed by 
W3C Members and other interested parties and has been endorsed by the Director as 

30 aW3C Recommendation, For further details on the DOM version L0 standard reference 
is made to the W3C website HTTP://www.w3.org/TR7l998/REC-DOM-level-l- 
19981000L , 
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As mentioned, the DesOM requires extensions to the DOM; These extensions are 
in the form of additional interface specifications. These specifications are detailed in this 
Section using the Object Management Group (OMG) Interface Definition Language 
(IDL). The specified interface represents a minimal interface for the DesOM. 
5 4.1.1 Interface Descriptor ! 

The Descriptor node object in the DesOM is a subclass of the DOM Element node 
object (see Appendix K). Like the Element node object, the Descriptor node object 
represents both the Descriptor element, as well as any contained elements. 



IDL Definition 



10 


interface Descriptor : 


Element { j 




void 


setHandler(in DescriptorHandler handler); 




DescriptorHandler 


getHandlerO; 




Nodelterator 

}; 


getSuperElementsO; 



Method setHandlerQ 

Set the DescriptorHandler for this Descriptor node objecjt This handler can be 
instantiated on the basis of the handler ENTITY that is specified as the value of 
the handler attribute for the Descriptor element. ; 
20 Parameters 

handler The DescriptorHandler to be assigned to this Descriptor node. 

Returns void j 

Exceptions This method throws no exceptions. 

25 Method getHandlerQ 

Returns the DescriptorHandler for this Descriptor node object. 

j 

Parameters None j 
Returns DescriptorHandler for the Descriptor node| object. 

Exceptions This method throws no exceptions. 

30 

Method getSuperElementsO 

i 

i 
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10 



15 



20 



None 

Nodelterator 

This method throws no exceptions, 



Returns a list of Descriptor generalisations or superElepients for the Descriptor 

! 

node object. j 
Parameters 
Returns 
Exceptions 
4.1,2 Interface DescriptorHandler 

The DescriptorHandler object provides methods for a class of Descriptor nodes. A 
DescriptorHandler can provide methods for more than one type of Descriptor, For 
example, a collection of Descriptors might use the same similarity metric. 

Preferably, the interface for the DescriptorHandler is fixexj. In other embodiments 
this interface can be specified either for a Descriptor or description scheme. 

The methods of a DescriptorHandler are generally implemented as class (static) 
methods. ; 

i 

IDL Definition I 



interface DescriptorHandler { 



boolean 
void 
void 
double 

); 



canCreateDescriptorContentO; 

createDescriptorContent(Descriptor descriptor, Entity resource); 
removeDescriptorContent(Descriptor descriptor) \ 
getSimilarity(Descriptor descriptorl, Descjriptor descriptor2); 



25 



30 



Method canCreateDescriptorContentQ 

Returns true if the DescriptorHandler contains an implemented method that can 
create the content for a descriptor.. 

I 

Returns True if a method has been implemented elke returns false. 



Descriptor node object 



Method createDescriptorContentQ 

Generates the content (ie. 5 child nodes) of the specified 
using the specified resource. 
Parameters 

descriptor The Descriptor node object for which the content (ie., child nodes) 
is to be created from the resource. , 
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resource The resource, represented as an entity, fro^n which the content is to 

be derived. ! 
Returns void j 

Exceptions This method throws a ResourceNotFoundfexception if the resource 
could not be found, or a HlegalResourceE^ception if the resource is 
not compatible with the method. I 



10 



15 



Method removeDescriptorContentO 



I 



Removes the content (ie. ? child nodes) of the specifiedj Descriptor node. This 
method might be invoked to reduce the complexity of a description for storage and 

i 

would typically only be invoked if the DescriptorHanjiler was capable of re- 
creating the specified descriptor's content. j 

i 

Parameters j 

i 

descriptor The Descriptor node object for which the content (ie., child nodes) 

is to be removed. 
Returns void 



20 



25 



30 



Method getSimilarityQ 

Returns a similarity metric in the range of [0, 1.0] which provides a measure of the 
similarity between the two specified Descriptor node objects. 

Parameters j 

i 

The first of the two Descriptor node objects to be compared. 



descriptorl 
descriptor^ 
Returns 
Exceptions 



The second of the two Descriptor node objects to be comparedv 



double 

This method throws an UnmatchedDescri^torException if the two 
Descriptor node objects are of incompatibly types. 
4-1.3 Interface AtomicDescriptorValue 

The AtomicDescriptorNode object is a subclass of the T6xt (node) object that is 

I 

specified as part of the DOM [The Text object contains the noh-markup content of an 

i 

Element]. It provides additional methods to the Text object which interpret the string data 
content of the Text object as other data types (ie., it is effectively : a typed text node). The 
data types available are as specified for the dataType attribute of the Descriptor element 
(see Section 3.1 .2.1). It is assumed in this specification that the XML data types (ie„ IDs, 
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IDREFs, ENTITY, ENTITIES) would be interpreted from ithe string value of the 
AtomicDescriptorValue node. < 

Dates and times are represented using the date an time formats specified by the 
profile of ISO 8601. Implementations of the AtomicDescriptorValue object can provide 
further methods that provide extra date functions (eg.;, getDataAsDateYearO, 

gefDataAsDateMonthO, etc.)* ! 

i 

IDL Definition ! 
interface AtomicDescriptorValue : Text { " j 



int 


getDataAsIntO; 


float 


getDataAsFloatO; 


double 


getDataAsDouble(); 


Date 


getDataAsDate() ; 


Time 


getDataAsTimeQ; 



}; 



Method getDataAsIntO 

Returns the value of the Text node as an integer. 
Parameters None j 
Returns Integer j 

Exceptions This method throws a DDFDataFormatEiception if the character 

i 

string could not be parsed as an integer. ! 

Method getDataAsFloatO ' 
Returns the value of the Text node as a float value. ' 
Parameters None ; 
Returns Float j 

Exceptions This method throws a DDFDataFormatException if the character 
string could not be parsed as a float value. 



Method getDataAsDoubteO 

Returns the value of the Text node as a double value 
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Parameters None 

i 

Returns Double 

i 

Exceptions This method throws a DDFDataFormatEpcception if the character 
string could not be parsed as a double valine. 

Method getDataAsDateQ 

Returns the value of the Text node as an ISO 8601 date. \ 

Parameters None ! 

i 

Returns ISO 8601 date j 

Exceptions This method throws a DDFDataFormatException if the character 
string could not be parsed as an ISO 8601 date. 
Method getDataAsTimeQ 

Returns the value of the Text node as an ISO 8601 time. 
Parameters None I 

Returns ISO 8601 time ! 

i 

Exceptions This method throws a DDFDataFormatException if the character 
string could not be parsed as an ISO 8601 time. 
5. Example of a Description Scheme 

An example of a description scheme expressed in DDF is contained in Appendix B. 
The description scheme aims to provide a description for digital video footage of an 

Australian Football League (AFL) game. This description scheme makes use.pf some 

i 

core element definitions that are contained in Appendix A, The (Tore.ddf is declared as an 
internal parameter entity Bl and then included in the description scheme using the % 
operator (see B2). The indicated lines Bl and B2 of the description scheme result in all 
the element definitions included in Appendix A being available to the example 
description scheme, j 

! 
I 

In the definition of the descriptor AFLGameDescription B3 a descriptor handler B4 
is specified. In this example, the descriptor handler is implemented as a Java class 
(AFLGameGen-class in the example contained in Appendix B) : having a predetermined 
procedural method which automatically generates the (description) content for the 
AFLGameDescription descriptor by analysing the digital video signal containing the 

footage of the game being described. ; 

i 
i 

j 
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It should be noted that although the AFLGameDescription element is defined as a 
specialisation of a Description element, a Description element is j just a specialisation of a 
Descriptor element, and so the AFLGameDescription can also be 'treated as a Descriptor. 

An example description generated from the description scheme contained in 
Appendix B is shown in Appendix C This example description would typically have 
been initially generated by the descriptor handler for the AFLGameDescription descriptor, 
however manual creation is also possible if an aimotator so desires. The procedural 
method to generate the content for the descriptor AFLGameDescription would typically 
analyse the digital video resource signal containing the footkge of the game to be 
described, identify the start and end of the four quarters of play,, and within each quarter 
track and, if possible, identify individual tracked players. The tracking could be achieved 
using motion analysis of the digital video resource with plaiyer identification being 
achieved by attempting to recognise a player's number from hi^/her jersey. It is not an 
object of this invention to specify a method for generating the content of the description. 

Clearly it is unlikely that all the information required j for the description, as 
specified by the description scheme, could be automatically generated from an analysis of 
the digital video resource signal. Where information is not available (eg., date and 
location of the game), the content generation method can; either generate empty 
descriptors or simply omit the descriptors from the description. At a later date an 
aimotator can add this information manually if it is required. Similarly, it might be too 
difficult for an automatic analysis to classify the action of eajch tracked player. For 
example, it might be difficult to automatically analyse whether the player was involved in 
a mark, a kick or a tackle. This information could also be provided at a later date. In fact, 
an aimotator could use a Digital Video Browser System, as described in Section 13, to 
browse the digital video resource and annotate as required. On completion of annotation 
the Digital Video Browser System could also be used to select to play all those sections of 

the digital video resource in which a particular player was involved, or all those sections 

i 

in which a mark occurred. In other words, the Digital Video Browser System could be 
used to complete any annotation tasks and browse the described digital video resource. 

Another example of a method to create the content for a descriptor, is one where the 
resource to be described has already been described using another description scheme. 
For example, a digital video camera might generate a description (using, for example, a 
Video Capture Description Scheme) for a digital video resourc^ as it is being captured. 
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The automatically generated description might contain information such as exposure, 
focus, eye-gaze location, shot boundaries, etc. It might be desirable to maintain some, if 
not all, of the information automatically recorded using the source description scheme, 
however it might be preferable to describe the digital video resoijirce using another more 
5 generally accepted description scheme, in this case the destination description scheme. In 

i 

this case the descriptor handler(s) in the destination description scheme could provide a 
mapping of descriptors from the source to destination descriptionjs. This mapping would 
typically be provided in the content creation method of despriptor handler for the 
Description element of the destination description scheme. This transformation from one 

10 . description scheme to another could also be achieved by applying rules to the DesOM 
(see Section 7). ; 
6. Methods of Applying Procedures . i 
6.1 Method of Generating Descriptions of Electronically- Accessible Resources 

Turning now to Fig. 7 A, there is a shown a method of generating descriptions of an 

15 electronically accessible resource. The method commences at step 700A and continues at 
step 702 A where a description scheme is read by a processor (i.e. the description 
generator). In the next step 704A, a processor identifies the one or more 
DescriptorHandlers in the description scheme and afterwards the method continues to step 
706A. In step 706A, the processor identifies the procedure^ corresponding to the 

20 previously identified DescriptorHandlers. These procedures are ih the form of procedural 
code contained in the DescriptorHandlers. In the next step 708A the proce<Jures are 
applied to the resource. The procedure generates a representative value which is 

associated with an attribute (i.e. feature) of the resource. The metjhod then outputs at step 

i 

71 OA the results of the application of the procedures. The method terminates at step 
25 712A. Preferably, these procedures result in the automatic generation of a description of 
the resource in the form of a DesOM which may be subsequently serialised as a XML 
document. However other procedures or processes may be envisaged. Further this 
resultant description is preferably interpretable by both humans arid machines. 
6*2 Methods of Applying Procedures to a Description 
30 Turning now to Fig. 7B, there is shown a flow diagram of a method of applying 

procedures to description(s) of resource(s). The method commences at step 700B and 
continues at step 702B where a description is parsed by a DDF processor. In the next step 
704B 7 the DDF processor identifies within the associated description scheme one or more 
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DescriptorHandlers. In the next step 706B of the method, the DFF processor identifies 

i 

the one or more procedures associated with the previously identified DescriptorHandlers. 
These procedures are in the form of procedural code contained in the DescriptorHandleis. 
In the next step 708B the procedures are applied to the DesOM corresponding to the 
description. The method then outputs at step 71 OB the results 6f the application of the 
procedures. The method terminates at step 712B. The method envisages many different 
types of procedures that can used in the method. In one embodiment, the method 
computes the similarity between two descriptors of the same ty^e. In this embodiment, 
the descriptions are parsed by the DDF processor and a commoi descriptor definition is 

i 

identified by the processor. The DDF processor then identified within the description 

scheme containing the common descriptor definition an associated DescriptorHandler 

which contains procedural code for computing similarity between two descriptors. The 

method then applies the procedural code to the DesOMs associated with the descriptions 

and determines the similarity of the descriptors and hence the similarity of the two 

15 resources. The method then outputs the results of the similarity computation. This 

i 

embodiment has particular application in searching/querying desdriptions of resources. In 
another embodiment, the procedural code of the method can encode and/or decode one or 
more descriptor components of the description of a resource. : This embodiment has 
particular application for efficient and/or secure transport or storage of descriptor 
20 components of descriptions of resources. i 

6,3 Examples of Methods of Generating Descriptions and Abplying Procedures to 
Descriptions 

The method of generating descriptions and applying procedures standardises the 
way descriptors and description schemes are defined. These descriptors and description 

25 schemes can be used to describe various types of multimedia dnformation* Using the 
descriptors and description schemes, descriptions that allow fast and efficient searching 
can be created and associated with multimedia content. Th6 preferred embodiment 
provides for automatic extraction of descriptors. However, jn general, this is only 
possible for low-level features. Features that represent higher le^el of abstraction usually 

30 have to be set manually or, at least, semi-automatically 

The method also provides a standard mechanism for associating descriptors with 
procedural code that generate them, which can greatly facilitate the deployment of the 
description schemes. For instance, such association will allow the development of very 
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general applications such as a multimedia database server, that make use of the procedural 
code to generate descriptors for new descriptions or for wmparijng descriptions. Apart 
from procedures for generating descriptors, procedures for \ validating descriptors, 
computing the similarity between two descriptors of the same type as well as encoding 
5 and decoding descriptors can be made available to applications through a standard 
interface. 

For example, the following exemplary applications are possible utilising such 

i 

procedural code. One can ■ 

• whistle a melody to find a song 5 j 

10 • play a few notes on a keyboard and get in return a list of musical pieces, 

• draw a few lines on a screen and get in return a set of images containing similar 
graphics, logos, ideograms, 

• define objects, including colour patches or textures and get in return examples, 

• using an exceipt of Pavarotti's voice, and getting a list of Pavajrotti's record, etc. 

15 The above scenarios involve the user providing some example content with his/her 

query. Standardisation of description schemes (in addition to a language for the exposure 
of DSS) would facilitate querying over multiple remote multimedia databases. 

There a number of problems relating to the standardisation of descriptors and 
description schemes. For example, there is a problem with evdn the relatively simple 

20 colour histogram descriptor. Even if two description providerk use the same colour 
histogram descriptor, they might use it in a different manner ^uch as using ilifferent 
quantisation. This will mean that histogram bin i may mean different things to different 
description providers. When one uses the histogram of an image as an example to search 
multiple image databases, one either has to compare and/or convert between different 

25 histograms. Both of these alternatives are difficult to achieve and error prone. It is also 
not practical to have every database server to re-compute the histogram of its images in 

the same way the example histogram was generated. j 

i 

The inventors propose two possible approaches for standardisation of description 
schemes: 

30 1. Standardise completely, to the last detail, the colour space, the colour quantisation 
(bins) for colour histograms, and consequently the matrix of cross-bin similarities. 

2. Use the image itself as the example. Then, each database uses its own extraction 

i 
i 
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method to compute the histogram of the received query image and then compare the 

i 

histogram with the rest of its database. 

The first option is not very practical, as people will never agree with every detail of 
the histogram's specification. Option 2 is more practical and cleaner, for histogram and 
5 most of other image queries. Option 2 means that each database can use their own 
particular parameters for descriptors and also their own methods | of computing similarity 
between descriptions. 

Only a low-level descriptor - the colour histogram - that coiild be generated from the 
content was considered as an example in these approaches- In practice, the query might 

10 also contain textual description or keywords that the user input ^vhich can be mapped to 

i 

some high-level descriptors such as the photographer's name, tljie caption of the image, 
etc. 

As described previously, a base Descriptor class from wjhich all descriptors and 
descriptions are to be based is defined and a description is itreated as a compound 

15 descriptor. The base Descriptor class includes an attribute that allows the URI of a 
handler that implements the descriptor's procedures to be stored. It provides a standard 
mechanism for associating descriptors with procedural code. iThe handler is called a 
descriptor handler. A standard API (application program interface) for the descriptor 
handler is based on a DescriptorHandler class. The DescriptorHjandler provides methods 

20 for generating the content (or value) of the descriptor, createDescriptorContent0, and 
computing similarity between two descriptors of the same t>{pe 5 getSimilarity(), (see 
section 4). , 

An alternative embodiment of the DesOM interface is now described below. The 
detailed definitions of the DesOm interface of the DescriptorHandler class can be found m 

25 Code Definitions A. In short, the DesOM interface of the \ DescriptorHandler class 
specifies the following methods: 



ParameterList getParameterList(in string methodName) 

i 

for getting a list of parameters that are relevant to the specified method. 
String getParameter(in string parameterName) 

for getting the parameter value. 
Void setParameter(in string parameterName, in string parameterValue) 
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for setting the value of a parameter. ; 
Descriptor createDescriptorContent(URI resource) 

for creating the descriptor for the resource. ' 
Double getSimiliarity(in Descriptor descriptorlObject, in Descriptor 

descriptor20bject) 

for computing the similarity between two descriptors of the same type. 
Boolean validate(in Descriptor descriptorObject) 

for validating the content of the descriptor. 
ByteAiray encode(in Descriptor descriptorObject) j 

for encoding the descriptor for transmission or archive, j 
Descriptor decode(in ByteArray encodedString) 

for decoding an encoded descriptor. 
In addition, the Descriptor class provides the following methods: 



Descriptor parseDescriptorString(in string XMLString) j 

for parsing an XML formatted string into a descriptor obiject (Note 
that, in this case, the parsing only check for well-formedness). 

String getXML() ~ — 

for returning the XML serialisation of the descriptor inchiding its start 
and end tags. ! 



Different descriptor handlers can be implemented (usually iby different developers) 
5 for any descriptor with different trade-offs among performance, functionality and 
complexity. However, all descriptor handlers of a descriptor 1 must comply with the 
definition of the descriptor. That is, the descriptor handlers can jonly generate descriptor 
value that conforms to the definition of the descriptor. At the same time, the descriptor 
handler can assume that any input descriptor will conform tb the definition of the 
10 descriptor. 

A descriptor designer may assign a default descriptor handler to a descriptor. 
However, a user of the descriptor is free to choose another handler or none at all. 

Not all descriptors are required to have a descriptor handler. Indeed, many 

descriptors of higher level of abstraction are expected to be handler-less. Nevertheless> 

I 
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even low-level descriptors may not have a handler. For instance, while a handler may 

i 

exist for a histogram descriptor, we don't expect a handler would be required for a 
descriptor that holds the creation date of a document 

Even if a descriptor has a descriptor handler, a description is not required to use or 
5 reference the handler. In addition, different instances of the same descriptor class may 
refer to different descriptor handlers, For instance, due to the different characteristics of 
different classes of images, for each class of image a different handler with a more 
efficient segmentation algorithm is used for creating its region jdescriptors. Moreover, 
applications are not restricted or required to use the descriptor handler referred to by a 

10 descriptor instance. 1 

At the same time, not all descriptor handlers (which I are subclasses of the 
DescriptorHandler class) will override the default implementation of all the methods of 
the base DescriptionHandler class, that is, provide support for all methods of the base 
class. For instance, a validation method may be implemented tolcheck that an ISBN has 

15 the right format; however, no method is implemented to generlte the ISBN. Another 
example is that while a descriptor handler may support the getSratilarity() method of a 
certain descriptor for an non-electronically accessible resource, it would not support the 
corresponding createDescriptorContentO method. 

Having the description (or the default handlers in the description schemes) pointing 

I 

20 to the relevant descriptor handlers and a standard interface for tbejdescriptor handler make 
it possible to build very general applications. For instance, the daiabase server application 
in the above mentioned option 2 does not need to have a predefined set of procedures 
linked in. Indeed, as is explained below, all the description providers in the above option 
2 can use the same database server application despite the different set of optional 

25 descriptors and the different descriptor parameters they used. j 

j 
| 
i 

Code Definitions A 

IDL definitions of the DescriptorHandler interface: j 

30 //File: DescHdlr.idl 

//Descriptor Handler IDL 

i 

#ifhdef_DescHdlr_idl_ I 
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i 

#define _DescHdlr_idl_ 
#pragma prefix "canoa.com" 

#include <Descriptor.idl> . 
5 #include <URIidl> \ 

i 

module DescriptorHandler { j 

! 
j 

interface DescriptorHandler; j 
10 . 

typedef sequence<octet> ByteArray, ; 
typedef sequence<string> ParameterList; j 
typedef sequence<string> MetricList; ! 

I 

15 enum ExceptionType { rNVALED_METHOD_NAME, 

METHOD_NOT_SUPPORTED, ; 
INVALID_PARAMETER_NAME, I 
INVALID_PARAMETER_VALUE 5 | 
XML_NOT_WELLFORMED s " 

20 NO_ACCESS_PRIVlLEGE, ; 

RESOURCEJUNAV AIL ABLE, ' 

RESOURCE_NOT_FOUND, [ 

FORMAT NOT SUPPORTED, i 
~ - i 

READ ERROR, j 
25 WRITE_ERROR, j 

INVALID_DESCRIPTOR_CLASS, i 

INVALrD_DESCRlPTOR_ATTRIBUTE ; | 

lNVALED_DESCRIPTOR_CONTENT 1 ! 

INVALID_METRIC > 
30 INVALID_ENCODED_STRING, 

OUT_OF_MEMORY, 

UNKNOWN_EXCEPTION }; 

i 
i 

CFP1595AU IPR34-38_GRPL490568 I:\ELEC\C1SRA\IPR\IPR34-38_GRP I \49 05 6$ .doc:PWM 

i 



-61- 



10 



// Exceptions 

exception DescriptorHandlerException { 
ExceptionType error; 
wstring description; 



interface DescriptorHandler 



{ 



// Get the list of parameters used by the descriptor. 
ParameterList gctParameterListO 

raises (DescriptorHandlerException); 



15 



20 



25 



// Get/set the specified parameters, j 
string getParameter(in string parameterName) 

raises (DescriptorHandlerException); 
void setParameter(in string parameterName, in string parameterValue) 

raises (DescriptorHandlerException); 



// The method creates the content of the descriptor using ! 

// the specified resource based on the current value of the 

// parameter attributes. If not supported, a METHOD_NOT_SUPPORTED 

// exception is raised. 

Descriptor createDescriptorContent(in UR1 resource) 

raises (DescriptorHandlerException); j 



30 // The methods compute the similarity between the two 

// descriptors passed in. The similarity measure computed 

// is returned. If not supported, a METHOD_NOT_SUPPO^TED 

I 

// exception is raised. ! 
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10 



double getSimilarity(in Descriptor descriptor!. , 
in Descriptor descriptor2) 
raises (DescriptorHandlerException); 



// Get the list of metrics supported for computing similarity, ; 

// The first in the list is the default metric used, j 

i 

MetricList getSimilarityMetricsO | 
raises (DescriptorHandlerException); : 



// Set the metric to be used for computing similarity, 
string setSimilarityMetric(in string metricName) 
raises (DescriptorHandlerException); 



15 



// Validate the content of the descriptor. If the content 
// is valid, return true; otherwise, return false. The 
// default implementation always return true, 
20 boolean vaiidate(in Descriptor descriptor) 

raises (DescriptorHandlerException); 



// Encode (compress) the descriptor for transmission or ■ 
25 // for archiving. If not supported, a METHOD_NOT_SUPP(|)RTED 

// exception is raised. I 
ByteArray encode(in Descriptor descriptor) 
raises (DescriptorHandlerException); 

30 

// Decode (decompress) the encoded descriptor. If not 

// supported, a METHOD_NOT_SUPPORTED exception is .raised. 

Descriptor decode(in ByteAnray encodedStr) : 
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raises (DescriptorHandlerException); ' 

>i : 

" }; 

5 #endif ! 

// JDescHdlr_idl_ j 

j 

The data (value) of a descriptor (or feature) usually depcnlds on some parameters. 

j 

For instance, the data of the colour histogram will depend on thfe colour space u$ed and 
10 the quantisation. These parameters, in general, are also used by the corresponding 
descriptor handler if one exists. Methods are provided in the DedcriptorHandler interface 
for obtaining a list of relevant parameters and setting the value <j>f the parameters. Note 
that the settings of the parameters control the characteristics of k descriptor instance but 
are not related to the actual content the descriptor instance described, 
15 hi the light of the fact that the DDF is XML based, the parameters are specified as 

XML attributes and data (value) describing the resource (content) should be part of the 
content model. For instance* an instance of the colour histogram descriptor may look 

like: | 

i 

i 

20 <! — rgbHistogram: each bin (marked by the <frequency> ta^s pair) stores the 

number of pixels whose value is between (r, g, b) and (H-binSize-l ? g-t-binSize T l, 
b+binSize-l) inclusive. — > 

<rgbHistogram binSize="32"> j 
<frequency r="0" g="0" b="0">l 4009</frequency> j 
25 <ftequencyr="32" g="32"b="32">21015</frequency> j 

i 
j 

<frequency jH"224" g="224" b="224">12434</jfrequency> i 
</rgbHistogram> ' 

The bin size, which is a parameter of the histogram, and the smarting rgb value of each 
30 bin, which are parameters of the (frequency) bin, are specified! as XML attributes. In 
contrast, the bin frequency, which describes the number of occurrence of a range of rgb 
value in the content, appears as value of the content model- Nevertheless, the principle of 
using XML attributes for descriptor parameters can only be treatejd as a guideline for good 
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i 

descriptor design and cannot be verified by a DDF processor. j 

As is evident from the interface defined for the DescriptorHsndler class, a descriptor 
handler may be used in the automatic creation of low-level descriptions, generate example 
descriptors for searching database, computing similarity between descriptors of the same 
5 class, validating descriptor content, and encoding and decoding descriptor content. 

Many low-level descriptors can be and, indeed, are expected Ito be extracted from the 
content automatically. It is even expected that some low-level descriptors can be created 
real time as the content is being captured. For instance, during the recording of a video or 
in subsequent processing, a descriptor handler for some generic video segment description 

i 

10 scheme can use the metadata provided by the video earner^ to segment the video 
temporally into clips (segments) and generate a description describing the structure of the 
video. Note that descriptor handlers of a non-standardised description scheme could also 
be used. For example, Fig. 8 shows a video processing application 808 generating a video 
segment description 802 utilising video and camera meta data 80|4. The video processing 

15 application 808 uses a video segment descriptor handler 800 from a standard library 806 

in generating the description 802, As can be seen, the description 802 refers to the 

i 

descriptor handler 800 . j 

When generating the description, the processing application calls the following 
methods of the descriptor handler: 
20 the getParametersO method to get the list of relevant parameters, 

the setParameterValueO method to set the parameters required, and 

the createDescriptorContentO method to generate the description 

i 

If required, the application may call the getXML() method of the descriptor to get the 
XML serialisation of the descriptor node. It is expected that otbjer structural components 
25 of the video that are difficult to extract automatically and higher level descriptors that 
describe the semantics of the structural components would be added to the description 
later with the aid of interactive tools. i 

The descriptor handler approach also allows developers to develop different extraction 
algorithms for generating descriptions for (low-level) features, 'and market or distribute 
30 them as some sort of "plug-ins". ; 

individual database servers can use the same set of descriptor handlers referred to by 
the descriptions it stored to generate similar or compatible descriptors for any example 
object specified in the query. A database server can then use] the descriptor handlers* 
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getSimilarityO method to compare the descriptors of the example bbject with those of the 

stored descriptions. For instance, in the above option 2 5 the client can send an example 

i 

image with its query to multiple remote image databases, E^tch database will then 
generate a histogram descriptor of the image using the descriptor handler referred to by 
5 the descriptions of its images and the same parameter settings usejd by the descriptions of 
its images. f 

For example, Fig. 9 shows how descriptor handlers can be usbd to support query-by- 
example searches over multiple remote image databases. A client 900 sends an example 
image 902 with its query to description/content providers A to Z. Each description/content 
10 provider A to Z comprises a image database 904 for storing! images, a description 
database 906 for storing colour histograms of the stored imagesj and a database search 
engine 908. The description/content provider A, upon receipt pf the query, generates 
91 OA a corresponding histogram descriptor 911 A utilising the colour histogram handler 

i 

912A referred to by the image colour histograms 914A stored in |its description database 
15 906. The description/content providers B to Z generate corresponding histogram 
descriptors 911B 5 ...,911Z in a similar manner. Namely, eac& provider generates a 
histogram descriptor of the example image using the descriptor handler referred to by the 
descriptions of its images and the same parameter settings used its descriptions. The 
provider A then computes the similarity of the example histogram 91 1 A with the image 
20 colour histograms 914A stored in its description database 906. : The providers B to Z 
compute the similarity of the example histograms 911B>..,911Z \vith the corresponding 
image colour histograms 914B,...,914Z in a similar mariner. '■ Those images and/or 
descriptions having a similar colour histogram are then retrieved from the databases 904, 
906 and are transmitted by the providers A,...,Z as query results tt the client 900. In this 
25 way, each provider may use different procedures for generating colour histograms, but at 

the same time provide consistent query results. The descriptor; handler approach also 

i 

allows a single database to use histograms with different parameters for the different 
classes of images it stores. 

Descriptor handlers also provide a flexible mechanism for computing similarities 
30 between two descriptors of the same class. The simple interface of the getSimilarityO 
method hides the complexity in computing the similarity between two descriptors. It 
allows the use of an appropriate algorithm and similarity metric for each class of 
descriptor and takes into account the different parameters the descriptors used (such as the 
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different bin sizes used by two rgb histograms). 

The descriptor handler also provides a way of validating descriptor content. It is 
possible in the serialisation syntax of the DDF (or an equivalent description definition 
language) to support the declaration of constraints. However, such a declarative approach 
5 is only possible for simple constraints such as such as maximunj value, minimum value, 
etc. Alternatively, the serialisation syntax can support the use of jan object model such as 
DOM and script language such as ECMAScript for specifying complex constraints. 
However, procedural code is generally a more efficient way :for validating complex 
constraints, 

10 Descriptor handlers also allow a more flexible approach for encoding and 

decoding descriptions or particular descriptors in descriptions, j Instead of using a 
single encoding/decoding algorithm for the entire description, more efficient 
encoding/decoding mechanisms can be developed for individual descriptors that make 
use of the characteristics of the individual descriptor. These iiiechanisms could be 

15 made available through the encode0/decode() method of descriptor handlers. The 
encoding/decoding procedure of any standardised descriptor and jdescription schemes 
can be made available as methods of some descriptor handler library. 

Fig. 10 shows an example how descriptor handlers fnight be used for 

i 

encoding/decoding standardised descriptors. The processing applications 1004, 1006 

i 

20 of the description consumer A and description provider B make use of methods of the 
descriptor handlers 1002 of a standard library 1000 to decode 1010 and encode .1008 
(as well in the case of the provider) the descriptor instance 1012. ■ 

The basic methods of the descriptor handler can be divided into two types: one that 
requires the resource (the content to be described) to be accessible, and the other that 

25 doesn't. Only the createDescriptorContentQ belongs to the first type and requires the 
content as well as some packages for processing the content to jbe available. The other 
methods such as GetSimilarity()> validateQ. encode() and debode() only operate on 

descriptor instances and do not require the use of a special multimedia (handler) library. 

i 

In addition to it being inefficient to upload content to a remote site for description 
30 generation, security and privacy issues with regard to the content exist. Therefore, it is 
expected most descriptions will be generated locally on sites where the content is located. 
Even in the case depicted in section 3.2, each database serverj uses a local descriptor 
handler to generate descriptors for the query example. In tha case where content is 
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downloaded for description generation, the application will Still be using a local 
descriptor handler. Hence, as far as description generation is concerned, any descriptor 
handler used would be a local one. 

Java presents an ideal object-oriented language for implementing descriptor handlers 
5 because of its cross-platform properties, its growing support on a ftarge variety of devices 
of various sizes and its close tie to the Web through which most descriptions are expected 
to be delivered. Some concern has been expressed over the issues that Java applications 
are not as efficient as other compiled code and that most existing features extraction 
algorithms are not implemented in Java, The advent of Just-In-Tfime (JIT) compiler has 
10 greatly improved the performance of Java applications and j applets. In addition, 
descriptions are likely to be generated using local descriptor handlers. That is, the 

createDescriptorContentO method is typically invoked locally. Hcfnce, it is free to use any 

i 

locally installed multimedia library including non-Java library through the Java Native 
Interface. As for the other methods of the descriptor handlers, thqy deal with descriptions 

15 and not content. They are usually not as processing-intensive as the 
createDescriptorContentO method. Standard Java packages are usually sufficient for their 
purposes. Hence, they can have a pure Java implementation which remote sites can 
download for execution. Signed applets can be used to lift the severe constraints that are 
typically imposed on standard applets. For instance, an appropriately signed descriptor 

20 handler may be allowed to write to and read from a specific local directory. 

Fig. 1 1 shows an example of descriptor handlers implemented as Java appleis. The 

i 

content/description provider B comprises a content database 1102 S a description database 
1 104, a description generator 1112 and a description server 11 \o. Descriptor handlers 
1116 of standardised descriptors are included in a standard library 1106 while those of 

25 non-standardised descriptors 1114 are be available in other descriptor handler libraries 
1 108. The descriptor handlers of both libraries 1108 and 1116 aire implemented in Java. 
The descriptions stored on the description database 1 104 can be generated by invoking the 
standard and/or non-standard createDescriptorContentO methods from the other 
descriptor handler libraries 1108 (non-standard) and/or standard library 1106. The 

30 description server 1110 retrieves requested descriptions from the description database 
1104 and transmits them to the client A Before transmitting the descriptions, the 
description server 1110 may invoke encoding methods from tfre standard and/or non- 
standard libraries 1 106, 1 1 08. j 
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The content consumer A comprises a processing application 1150 and a standard 
library 1152. The processing application 1150 receives the encoded description and is 
decoded by invoking a decoding method from the standard library 1152. The decoded 
description forms a descriptor instance. Non-standard descriptor handler applets 1 1 14 
could be downloaded from the provider A, if required, and all their methods could be 
executed on the client machine A. In particular, a non-standard decoding descriptor 
handler can be downloaded 1154 to form a descriptor handler instance 1156, The 
processing application 1150 then invokes the non-standard decoding descriptor handler 
1156 to decode 1160 the encoded description 1199 to produce! the descriptor instance 

1158. ' 

The content/description provider B can use a Java Native Interface 11 20 as part of the 
Java packages and non-Java libraries 1122 of descriptor handlers. For instance the 
createDescriptorContentO may be implemented in Non-Java code and thus can only be 
invoked locally on the server machine B through the Java Native Ihterface 1 120. 

In summary, descriptor handlers could be implemented as applets. Descriptor 
handlers of the standardised descriptors can be provided as part of a standard library 
(together with the definitions of the standardised descriptors anjd description schemes). 
However, users are free to use any valid descriptor handlers. Descriptor handlers for non- 
standardised descriptors would be available separately in other libraries. It is proposed 
that descriptor handler applets are properly signed and| all methods of the 
DescriptorHandler interface except possibly createDescriptorContentO are expected to be 
downloadable to remote site for execution. The createDescriptorContentO method may 
require that special libraries or native libraries would cause exception when not invoked 
locally. I 
7. Rule-based Processing using the DesOM ! 

The internal memory structure of a description (ie., tfjie DesOM) provides a 
convenient structure on which to perform further processing of ja description (or indeed 
the relevant description scheme). This further processing can be achieved by locating 
patterns of nodes in the DesOM and performing specified actions in response to the 
located patterns. Each pattern-action association can be represented by a rule and a set of 
related rules can be collected into a rule set. 

Rules can be used to used to automatically create further descriptors based on 
existing descriptors (see Section 8. Method of Extending Descriptions of Resources), to 

C pp 1595AU IPR34.3S_QRP14<>0S6S I:\ELE(^CISRA\IPR\IPI^4-38^GRPl\4905(58.doc:PWM . 



-69- ; 

provide presentation properties for descriptions and description Schemes (see Section 9. 
Method of Presenting Descriptions of Resources), and to represent queries (see Section 
10. Method of Selecting Resource Descriptions ). Rules can also be used to translate a 
description to the language of the query (see Section 11. Method of Translating a 
5 Description of a Resource). The Digital Video Browser System (described in Section 13 
uses a method for formulating rules common for each of these functions. This method is 
described below, i 

Each rule consists of a pattern (of nodes in the DesOM) arid an associated one or 
more actions. For each of the different functions (inference, equivalence, presentation 

10 and selection), a different set of actions is often applicable. However each of these 
functions can be enabled using a common rule grammar which Vdll be described in this 
section. The rule grammar can be defined in an XML DTD. Ttje rules for the different 
functions can simply use the common rule grammar (this is the c2j.se for the Digital Video 
Browser System), or alternatively the allowable actions can be! controlled by defining 

15 different DTDs for each of the different functions (eg. 5 an InferenceRules.dtd, a 
PresentationRules.dtd, etc,)- j 

Rules can be represented as, or in a manner similar to, Extensible Style Language 
(XSL) rules. In the Digital Video Browser System (see Sectioiji 13), we have used the 
following basic rule grammar, 

I 
i 

i 
i 

i 
I 
i 

! 
i 
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Rules.dtd 



10 



15 



20 



25 



30 



<! ELEMENT Rule (Action+> 
<! ATTLIST Rule 

target (Element | ElexnentDefd) 
pattern CDATA 



"Element" | 
#REQUJRED| 



<! ELEMENT Action ( AddAttribute | RemoveAttribute j 

| AddElement | RemoveElement ! 

i 

| AddAttributeDef | RemoveAttributepef 
| Select)> j 
<!ELEMENT AddAttribute (EMPTY)> 
<! ATTLIST AddAttribute 
attName CDATA 
attValue CDATA 

> 

<! ELEMENT RemoveAttribute (EMPTY)> 
<! ATTLIST RemoveAttribute 



#REQUTRED| 
#REQUTREDj 



attName CDATA 



#REQUIRED 



<!ELEMENT AddElement(#PCDATA)> 
<! ATTLIST AddElement 

position (SiblingBefore | SiblingAfter | AsFirstChiltl | AsLastCbild ) 

#required| 



,1, 



<! ELEMENT RemoveElement (EMPTY)> ; 

<!ELEMENT AddAttributeDef (EMPTY)> j 

i 

<! ATTLIST AddAttributeDef j 

attName CDATA #REQUIRED 

attType CDATA #REQUIREI? 

attDefault CDATA #REQUIRED 

> i 
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<!ATTLIST RemoveAttributeDef 




attName CDATA 


#REQUIRED ; 

I 


> 

<1ELEMENT Select (EMPTY)> 


\ 


<!ATTLIST Select 




attName CDATA 


"selected" 

i 


attValue CDATA 


YES i 


selectAncestors (YES (NO) 


"YjES" 

! 


> 





Each Rule element has a target attribute that has a default value of "Element 1 ' and a 
character string pattern attribute. The target attribute refers to tlje target of the defined 
Rule. Typically inference, equivalence and search rules are targeted at elements because 
the action of the rule results in either a new descriptor in the description or the selection 
of a descriptor for a query. Presentation rules, however are typically targeted at element 
definitions as their associated actions specify how a particular 4escriptor type is to be 
presented in an application. A set of rules can be serialised in an XML document. This is 
typically the case with inference, equivalence and presentation j rules, but may not be 
required for selection rules which may often be processed on a single rule basis. 

The role of the pattern character data string is to identify th^ particular elements (or 
element definitions) to which the action is applied. This character string can identify 
more than one element and can include element ancestry and attribute qualifiers. 
Preferably, the pattern string is parsed according to the following Extended Backus-Naur 
Form (EBNF) notation. ! 



Pattern 


: : =ElementPatternsf ConnectorippElementPatterns)* 


ElementPattems 


ElementPattern (AncestrvOn ElementPattern)* 


ConnectorOp 


::= T | 


AncestryOp 


::= 7' | 7/' 



Each pattern can consist of one or more alternative patterns (ie., *|' represents an 
alternative) or must satisfy more than one ElementPattern (ie. s ( & 7 connector operation). 
Element ancestry is represented within a pattern by using the parent operator 7\ Two 
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patterns separated by a parent operator match an element if the right hand side matches 
the element and the left hand side matches the parent of the element. For example, the 
following Shot elements that have a Scene element *s a parent and a 
* VideoClipDescription element as a grandparent match the following Rule's pattern: 
5 <Rule pattern = "VideoClipDescription/Scene/Shot"> j 

<Action> etc . „ <J Action> ; 

</Rule> 1 

i 

Two patterns separated by the ancestry operator 7/' match jan element if the right- 
hand side that matches the element has at least one ancestor jthat the left-hand side 

10 matches. So, for example, any Shot elements that have a VideoClipDescription as an 

i 

ancestor element will match the following Rule's pattern: j 
<Rule pattern "VideoClipDescription//Shot"> ! 

<Action> etc . . .</ Action> j 
</Rule> j 

i 



i 
i 

i 

i 
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10 



15 



ElementPattexn 

ElementTypePattem 

OneElementTypePattern 

ElenientQualification 

Qualifiers 

Qualifier 



AttributeQualifier 
Attribute? attem 
AttributeValue 
PositionalQualifier 

Position 



:~ ElementTvpePattern ElemlentOualification 



20 



OneEleme ntTy pePattern j 
:= ElementTyneName 
:« '[' Qualifiers ?'!' 
:= Qualifier (V OualifierY * 
:= ChildOualifier 

) AttributeQualifier i 

I PositionalQualifier 

AttributePattern ( ' =* J Attribute Valued ? 
~ 'attribute' ^'AttributeNafne')' 

Position T 

i 

:= TirstOfType 1 | 'NotFirstbfType* 
| TirstOfAny' | 'NotFirstOfinany' 
| 'LastOfType' | 'NotLastOfType' 
[ 'LastOfAny 7 1 'NotLastp£\ny' 
j 'OnlyOfType' | 'NotO^yOfType' 
| 'OnlyOfAny' | 'NotOnlyOfAny' 



25 



30 



An element within the pattern hierarchy may have qualifiers applied to it, which 
further constrain which elements match the term. These qualifiers may constrain the 
element to have certain attributes or sub-elements or may constrain its position with 
respect to its siblings. The qualifiers are specified in square brackets following the 
ElementTypeName (which is it tag name defined in the DTD). X pattern matches only if 
all of the qualifiers are satisfied, \ 

For example, any Shot elements that have a child element keyFrame will match the 
following Rule's pattern: 

<Rule pattern = "Shot(XeyFrame]"> 
<Action> etc. . ,</Action> 

</Rule> ! 
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1 

Attributes on the target element or any of its ancestor elenients can also be used to 
determine whether a particular rule applies to an element. Ah attribute qualifier can 
constrain an element to have either a specific attribute with a specific value, or to have a 

specific attribute with any value. For example, the following pattern matches a Bin 

i 

5 descriptor which has as its parent a Histogram descriptor which has an attribute noBms 

with a value of L 100': I 

i 

<Rule pattern = "Histogram[at1ribute(noBins)='100 s ]/Bin"^ 
< Action> etc . . ,</Action> I 
</Rule> j 
10 Positional qualifiers can also be used to further constrain thje pattern to match on the 

element's position or uniqueness amongst its siblings. For jexample, the following 
example matches Object descriptors which are the only Objects iijt a KeyFrame descriptor: 

<Rule pattern - < *KeyFrame/Object[OnIyOfType()]"> j 

i 

<Action> etc . . .</Action> j 
15 </Rule> 

The above description of the matching method permits pjattem matching only on 

i 

elements (which are typically descriptors in the DesOM) or element definitions. Clearly 
there are many possible embodiments for defining the syntax of the node pattern matching 
without departing from the spirit and scope of the invention. 
20 Each Rule can have one or more associated Action elements. In the Digital Video 

Browser System (see Section 13) the allowable Action elements for rules has been limited 
to the addition and removal of elements and attributes from elements (ie, 3 descriptors) in 

descriptions and the addition and removal of attribute definitions jfrom element definitions 

j 

in a description scheme. The actions involving individual descrijptions are generally used 

i 

25 by inference, equivalence and selection rules (see Sections 8 land 10) and the actions 
involving description schemes are generally used by presentation jrules (see Section 9). 

The attributes of the Action elements, AddAttribute and RemoveAttribute, specify 

the attribute to be added or removed from a target element (jie., an element that has 

j 

matched the specified pattern in the rule). The content of the AddElement action contains 
30 the element to be added to the DesOM as a relation of a target element. The position 
attribute of the AddElement element specifies where the new element should be added 
with respect to the target element. This position attribute can indicate that the new 
element is to be added as a sibling node before the target element (SiblingBefore), as a 
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i 

sibling node after the target element (SiblingAfter), as the first ckld of the target element 
(AsFirstChild), or as the last child of the target element (AsLastfbhild). Clearly, since the 
element to be added to the DesOM is represented as parsed character data (#PCDATA), 

i 

an element hierarchy can also be added to the DesOM. The RemoveElement action will 
5 simply remove a target element. Any child elements of the tarjget element will also be 
removed. j 

The AddAttributeDef and RemoveAttributeDef actions are only valid if the target 
for the rule is an element definition. These actions are typically dsed by presentation rules 
(see Section 9). The AddAttributeDef action uses the attNameL attType and attDefault 
XO attributes to specify the required information for the attribute definition to be added to an 
element definition. The RemoveAttributeDef action will simply remove the attribute 

definition that is identified by the value of the attName attribute of the action. Attribute 

i 

definitions can be replaced by including both an AddAttributeDef and a 
RemoveAttributeDef action in a particular rule, i 

15 The Select action is typically only used by selection rules and is described in detail 

Section 10. Rules can also be used to transform a description. ; These rules are used to 
generate a second description conforming with a second description scheme. 
8. Method of Extending Descriptions of Resources i 

Given a description scheme, it is possible that further descriptors can be 

20 automatically created by inference or a known equivalence in a [description based on the 
existence or otherwise of a particular set of descriptors. For example, if a descriptor for a 
digitally captured image representing light exposure levels injiicated outdoor lighting 
levels, then an additional descriptor could be automatically created to classify the image 

as an "Outdoor Scene". Since the latter classification can be interred from the recorded 

i 

25 light exposure levels there is no advantage in storing the classification because it can 

always be re-generated while the inference rule exists. Rules can also be used to generate 

! 

textual descriptors based on non-textual descriptors or vice vjersa. For example, the 
colour of an object might be stored in a description as a (R, G, ^) value. A rule could be 
formulated which maps each (R, G, B) value to one of a possible number of colours 
30 represented in a text string (eg., red, green, purple, etc). The additional descriptors 
generated by inference or equivalence rules can result in a richer description that can be 
exploited by applications (eg, 7 search engines, filter agents, etc.).| 

! 
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i 
i 

i 

A set of rales that is applicable for a given description sjcheme can be serialised 

i 

(stored) in an XML document. In the Digital Video Browser System (see Section 13) , a 
reference to such an XML document is stored in the value of the ruleSets attribute of the 
Description element for the description scheme (see Section 3J.2.2 Description 
5 Definition). It is possible to associate more than one rule set with a description scheme. 
In the Digital Video Browser System ( see Section 13), if more than one rule set is 

specified then it is assumed that both rule sets can be applied (iel, the individual rule sets 

i 

do not contain unresolvable rules). In other words, the indivi4ual rule sets are simply 
combined and treated as a single rule set, in which the order of rules to be processed is 

10 provided by the order of the listing of the individual rule sets and the order of the 

i 

individual rules within each given rule set. Inference and equivalence rule sets can also 
be stored with an application without departing from the essence of the invention, 
however in this event the value of the rules is limited to the particLlar application. 

Preferably, the Action elements typically used are the ajddition and removal of 

15 attributes and elements from the DesOM. Replacement can be achieved by using a 

I 

removal followed by an addition Action element, i 

A set of inference rules is preferably invoked whenever a description is first 
processed into the DesOM. The rules are iteratively processed juntil no further changes 
can be made to the DesOM as some rules may depend on the actions of other rules. The 

i 

20 rule set may need to be (iteratively) reapplied whenever the description is updated (eg., a 
manual annotation in an application utilising the description) j In the event, ,that an 
application has permitted changes to be made to the description, then before serialising 
the altered description each change needs to be considered in light of the inference rules 
in order to ascertain whether the descriptor can be inferred from j knowledge of the other 

25 descriptors in the description. If a descriptor can be inferred the4 it is excluded from the 
serialised description. ! 

The method preferably associates a set of inference and/or equivalence rules to a 
description scheme. This set of rules can be implemented according to the 
abovementioned description and results in a richer description structure without any 

30 additional storage or transport overhead which would result if the extra (inferred or 
equivalent) descriptors were included as part of the individual descriptions. Being able to 

represent this inferred or equivalent information as a set of rules that can be invoked when 

i 

required represents a significant saving in storage and transport cost if a large digital 
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library were to be described. In other words it can eliminate the storage and processing 

costs of redundant information. [ 

An important aspect of the method is that unlike existing stylesheet languages such 

as XSL, the inference and equivalence rules do not form the basis of a construction of a 
5 new tree structure which is typically used for rendering. In tlje method the rules are 

applied to the memory structure that represents the description the DesOM) and result 

in changes to that structure. The role of the rules is to provide a richer description of the 

resource that can be exploited by applications (eg., search engines, filter agents, etc.). 

This richer description does not necessarily need to be serialised because the richer 
10 description can always be generated from the original description jusing the rules. 

The embodiment for applying the inference and equivalence rules has a limited set 

of actions that can be performed on the selected elements (seej Rules.dtd in Section 7. 

Rule-based Processing using the DesOM, This set of actions is sufficient for the Digital 

Video Browser System described in Section 13, however it is possible that a more 
15 extensive set of rules may be required for other applications. ! 

Turning now to Fig. 12, there is shown a flow diagram of ia method of_extending a 

i 

description of a resource. In step 1200, the method commenced and a host application 
such as a search engine invokes a DDF processor and selects a description in response to a 
user request for further processing. The description may be generated as prviously 

20 described. In the next step 1202, the DDF processor parses the description into a DesOM. 

i 

After step 1202 the method continues at step 1204 ? where an associated set ofxjules are 

accessed using the RuleSet attribute of the description scheme. Tfhese set of rules may be 

i 

serialised in the foim of an XML document. In the next step 120£, the first rule of the set 

i 

is selected for processing. 1 

i 

25 The method then continues to decision block 1206, where & check is made whether 

a pattern associated with the selected rule can be found in the DesOM. The manner in 
which the pattern associated with the selected rule matches a pattern in the DesOM is 
described in more detail in 7. Rule-based processing using the DesOM. If the decision 
block 1208 returns true(yes) ? then the processing continues at the next step 810, where the 

30 inference or equivalence action associated with the rule is initiated on the DesOM, These 
actions preferably initiate addition and removal of attributes and elements from the 
DesOM thus modifying the DesOM. Afterwards, the method selects the next rule in step 
1211 and the processing returns to decision block 1208. If the decision block 1208 
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returns false^o), the processing continues at decision block 1212, where a check is made 
whether all the selected rules have finally been processed withoujt action. In this way, the 
rules are iteratively processed until no further changes can be mide to the DesOM. This 
is advantageous in the situation where some rules are dependent on other rules. If, on the 
5 other hand, the decision block 1212 returns tnie(yes) ? the prodessing continues at step 
1216 where the extended DesOM is output The method then terminates at step 1218. 
9. Method of Presenting Descriptions of Resources 

A description could be used by many applications. Each application might exploit 
different properties of the description and its defining descriptiori scheme. Some of these 
10 applications will invariably need to represent description schemes and/or descriptors in a 
graphical or pictorial manner. For example, many descriptors could be graphically 

represented by icons and a user's interaction with either a description or description 

i 

scheme could be mediated by icon selection. \ 

Presentation properties for descriptors could be included a|s part of the description 

15 scheme however this can be non-ideal for two reasons. First, the role of the description 
scheme and description is to describe classes of resources anjd a particular resource, 
respectively, and it is preferable to keep both entities as concise! and precise as possible. 
Presentation information would result in extra presentation infonkiation (eg., icons) being 
part of a description scheme (and perhaps descriptions) and would therefore increase the 

20 storage and transmission costs for each description scheme. Second, different 
applications might prefer to present descriptions and description schemes in, different 
ways. In other words, the presentation properties of descriptions iand description schemes 
can be application dependent. , 

It is advantageous, however, to have a set of presentation rules grouped in a rule set 

25 that can be serialised, transported with and used in conjunction with the description 
scheme so that other applications can, if they choose to, use a similar set of presentation 
rules. This would not be the case if the presentation rules were tightly linked with a 
particular application (ie„ part of the application code base). i 

As with inference and equivalence rule sets, presentation rule sets canoptionally be 

30 linked with a description scheme by specifying the XML document containing the 
presentation rule set as the value or part of the value of the ruleSets attribute in the 
Description element for the description scheme (see Section 3.1 .2.2 Description 
Definition). Presentation rule sets can be included in the ruleSjets attribute along with 
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other rale sets that might be concerned with inference and equivalence rules. In the 
Digital Video Browser System, which is described in Section 13 3 the presentation rule sets 
are stored with the description scheme in the ruleSets attribute. Alternatively, they could 
be stored with the application rather than the description scheme. Presentation rule sets 
5 stored as part of the description scheme are processed like inference or equivalence rule 
sets. In other words, all the rules from the individual rule sets are combined into a single 
rule set. Resolution of rules is performed on the basis of rule order (as was described for 
inference rules in Section 8, Method of Extending Descriptions of Resources). If an 
alternative method of processing presentation rule set(s) is required then the presentation 

10 rule $et(s) are best stored with the application so the application can control the 
processing. i 

Presentation properties can be attributed to the descriptor definitions in a description 
scheme or the descriptor elements of a description using application-specific presentation 
rules. Unlike, inference or equivalence rules, a presentation rule is typically applied to an 

15 element definition in a DTD. Its role is to provide presentation behaviour for the 
instances of the descriptors defined in the description scheme. In the Digital Video 
Browser System (see Section 13), presentation rules are only applied to descriptor 
definitions and not to descriptors within individual descriptions. However, it is 
conceivable that some applications might benefit from an ability to define presentation 

20 rules based on individual descriptors in descriptions. The rules in a presentation rule set 
can be formulated in a similar way to inference or equivalence rule sets. 

Preferably, the Action elements of presentation rules typically involve the addition 
and removal of attribute definitions in element definitions (in the description scheme). 
Consequently the rules are targeted at element definitions i rather than elements. 

25 Alternative embodiments could apply presentation rules to individual descriptions and 
therefore the target of these rules would be elements rather than element definitions. 

Presentation rules arc used in the Digital Video Browser System described in 
Section 13 for the following functions: 

• To classify descriptors as being structural (hence belonging in a Table of Contents) or 
30 of an index nature (hence belonging to an Index); 
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• To assign icons to descriptors where the icons are assigned on a description scheme 
basis (ie., by the addition of attribute definitions having default values to descriptor 
definitions), and; 

• To add "Selected" attributes to all selectable descriptor definitions so that selection 
rules can interact with the presentation of the descriptions (eg., st> the application can 
differentiate visually between selected and non-selected descriptors). 

The method involves associating a set of rules with a description scheme that can 
influence the presentation properties of descriptors in descriptions which are conformant 
with a particular description scheme. It is an advantage to have these presentation rules 
grouped in a rule set that is either linked to a description scheme so) that applications can 
utilise the defined set of presentation properties if required. Alternatively an application 
can select to use its own set of presentation rules. 

Turning now to Fig. 13, there is shown a flow diagram of a method ofjvisually 
presenting a description of a resource. In step 1300, the method commences and a host 
application such as a search engine invokes a DDF processor. In the next step 1301, a 
description is selected for presentation. This selection can occur bjy way of user input or 
by way of another application. The method then continues at istep 1302, where the 
associated defining description scheme is read into memory. The; description scheme in 
memory comprises an array of element definitions where each element definition has an 
airay of attribute definitions. Alternatively, the DDF processor can parse the description 
into a DesOM. After step 1302 the method continues at step 1304 y where the presentation 
set of rules are accessed using the RuleSet Attribute of the description. In the next step 
1306, the first presentation rale of the set is selected for processing. 

The method then continues to decision block 1308, where ai check is made whether 
a pattern associated with the selected rule can be found in the DesOM. A pattern matching 
process similar to that described in 7. Rule-based processing using the DesOM would be 
suitable. If the decision block 1308 returns true(yes), then the processing continues at the 
next step 1310, where the attribute definition(s) associated with the rule is removed or 
added to the array in memory. Afterwards, the method seJects the next rule in step 1311 
and the processing returns to decision block 1308. If the decision block 1308 returns 
false(no) J the processing continues at decision block 1312, where a check is made whether 
all the selected rules have finally been processed without action. In this way, the rules are 
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iteratively processed until no further changes can be made to the array in memory. This is 
advantageous in the situation where some rules are dependent on other rules. If, on the 
other hand, the decision block 1312 returns true(yes), the processing continues at step 
1316 wherein a modified description is created using said modified description scheme as 
a template. This modified description is then output to an output device. For example, 
the modified description and it's associated resources, such as digitjal video resources or 
DVDs, can be rendered on a display or a printing device* ' 
10. Method of Selecting Resource Descriptions 

Selection rules can be used to formulate queries directed at collections of 
descriptions (eg., digital libraries). A query can be viewed as a rbquest to select those 
descriptions or components of descriptions (ie., descriptors) that match a specified 
pattern. Like inference and equivalence rules, selection rules are typically directed at 
elements rather than element definitions. Unlike inference, equivalence and presentation 
rules, however, selection rules may be generated on a one-off basp and not collected in 
rule sets that are serialised in an XML document. For example, a query is usually 
formulated with help from the user, then processed, and the resulfe presented to the user 
for their evaluation. 

Selection rules often depend on presentation rules in that the selection action must 
be able to be interpreted by the application and presented to the user. For example, a 
selection action could simply set a (presentation) attribute for descriptors that match the 
specified pattern. 

Selection rules are typically associated with the application. In the Digital Video 
Browser System (see section 13), selection rules use the same grammar as all other rules 
(see Section 7. Rule-based Processing using the DesOM)- However, typically the only 
Action that is invoked by a selection rule is the Select action. Consequently it would be 
possible to define a more specific grammar for selection rules i(eg., SelectionRules.dtd 
having just a Select action being allowed). 

The Select action of a selection rule has three attributes! which specify how the 
selection action is implemented- The value of the attribute attName refers to the attribute 
name used for a descriptor that is able to represent the action ;of being selected. This 
attribute would typically have been generated using a presentation rule. If the element 
matched by the pattern does not contain such an attribute, then the selection process will 
search for ancestors of the matched element in the DesOM (ie.j, up the description tree) 
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until it locates an element with the specified attribute name. In the above DTD this 
attribute name is provided with a default value of "selected". Th$ value of the second 
attribute attValue refers to the value that the "selected" attribute should be assigned in 
order to indicate selection. The DTD also provides a default value of "YES" The third 
5 attribute specifies whether all selectable ancestors should also be selected. So, for 
example, if a user selects a Shot descriptor because of a matched descriptor contained in 
the Shot descriptor, then the user should also select the ancestors of the Shot descriptor 
(ie, the Scene descriptor and the VideoClipDescription descriptor). 

In this way, the Select element provides information to the application on which 
10 elements have matched the specified pattern in the selection rule. Clearly the application 



i, hence the interaction 
Browser System (see 



needs to be aware of the attribute used to provide this information, 

between presentation and selection rules. In the Digital Video; 

Section 13), selection rules are used to implement searches in a Digital Video Library, 

The method involves that of representing queries by selection rules which attempt to 
15 find matches to a rule's specified element pattern. The "select" action that is executed on 

a successful pattern match typically modifies attributes established by presentation rules, 

so that the selection process can interact with the application. 

Turning now to Fig. 14, there is shown a flow diagram of a method of selecting one 

or more descriptions or part of one or more descriptions of a resource. In step 1400, the 
20 method commences and a host application such as a search engine invokes a DDF 

processor. In the next step 1402, a user inputs a query which is." formulated as a rule in 

step 1404. The search engine then selects in step 1405 a first description for evaluation. 

The method then continues at step 1406, where the DDF processor parses the description 

into a DesOM. 

25 The method then continues to decision block 1408, where a check is made whether 

a pattern associated with the selected rule can be found in the DesOM. The manner in 
which the pattern associated with the selected rule matches a pattern in the DesOM is 
described in more detail in 7. Rule-based processing using the DesOM. If the decision 
block 1408 returns true(yes)> then the processing continues at the next step 1410, where 

30 the select action associated with the rule is initiated on the DesOM. The details of the 
select action is described above. Afterwards, the method then continues at decision block 
1412 where a check is made whether the last description has been searched. If the 
decision block returns false(no) the processing continues at step 1414 where the next 
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description is selected. Otherwise, the processing continues at step 1416, where the 
results of the searching process is output. The method then terminates at step 1418. 
1 1 . Method of Translating Descriptions of Resources 

Often descriptions of resources will be in a language different from the request. 
Rather than store copies of the descriptions in each language, the rriethod stores only one 
copy of the descriptions in one language. Preferably, the language is English. The method 
is then provided with a number of rule sets that enable the translation of the descriptions 
to the language of the request. For example, the description may have a "colour" 
attribute and a colour attribute value "red". If the request is received in French, then the 
method will translate the description to French. In the example giyen, "colour" and "red" 
will be translated to their French equivalent. This is a form of interf language equivalence. 
This procedure is similar to the way Inference Rules are processed, but on a conditional 
basis. Inference rules are preferably not processed on a conditional basis as described 
here for translation rules. 

Turning now to Fig. 15, there is shown a flow diagram of a method of translating a 
description of a resource. In step 1500, the method commences) and a host application 
such as a search engine invokes a DDF processor and selects a description in response to a 
user request for further processing. In the next step 1502, the DDF processor parses the 
description into a DesOM. After step 1502 the method continues at decision block 1503, 
where a check is made whether the language of the request is different from the language 
of the description. This check is accomplished by comparing thfe language attributes of 
both the request and the description. ' 

If the decision block 1503 returns true(yes), the processing continues at step 1504, 
where an associated translation set of rules are accessed using the RuleSet Attribute of the 
description. These translation set of rules may be serialised in the form of an XML 
document. On the other hand, if the decision block returns false(no) then the processing 
continues at step 1516. After completion of step 1504, the method continues at step 1506, 
where the first rule of the set is selected for processing. i 

The method then continues to decision block 1506, where: a check is made whether 
a pattern associated with the selected rule can be found in the DesOM. The manner in 
which the pattern associated with the selected rule matches a pattern in the DesOM is 
described in more detail in 7. Rule-based processing using the DesOM. If the decision 
block 1508 returns true(yes), then the processing continues at the next step 1510, where 
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the translation action associated with the rule is initiated on the DesOM These actions 
initiate the removal and addition of attributes and elements from the DesOM. The 
removal and addition action substitutes the language of the attributes and elements for 
another. Afterwards, the method selects the next rule in step 1507 and the processing 
returns to decision block 1508. If the decision block 1508 returns false(no) s the 
processing continues at decision block 1512, where a check is made whether all the 
selected rules have finally been processed without action. If, on the other hand, the 
decision block 1512 returns true(yes), the processing continues at step 1516 where the 
extended DesOM is output. The method then terminates at step 1518- Alternatively it is 
also possible to include an action of a rule which invokes a DescriptorHandler method to 
translate the content of the selected Descriptor. 
12. First Embodiment of Apparatus 

The processes described in relation to Figs. 1A to 15 can be practiced using a 
conventional general-purpose computer, such as the one shown in Fig. 19 wherein the 
processes may be implemented as software executing on the computer. In particular, the 
method steps are effected by instructions in the software that are carried out by the 
computer. The software may be divided into two separate parts; one part for carrying out 
the processing steps; and another part to manage the user interface between the latter and 
the user. The software may be stored in a computer readable medium, including the 
storage devices described below, for example. The software is loaded into the computer 

from the computer readable medium, and then executed by the computer. A computer 

i 

readable medium having such software or computer program recorded on it is a computer 
program product. The use of the computer program product in the computer preferably 
effects an advantageous apparatus in accordance with the embodiments of the invention. 

The computer system 1900 consists of the computer 1902, a video display 1 916, and 
input devices 1918, 1920, In addition, the computer system 1900 can have any of a 
number of other output devices including line printers, laser printers, plotters, and other 
reproduction devices connected to the computer 1902. The computer system 1 900 can be 
connected to one or more other computers via a communication interface 1908b using an 
appropriate communication channel 1930 such as a modem communications path, a 
computer network, or the like. The computer network may include a local area network 
(LAN), a wide area network (WAN), an Intranet, and/or the Internet 
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The computer 1902 itself consists of a central processing unil(s) (simply referred to 
as a processor hereinafter) 1904, a memory 1906 which may include random access 
memory (RAM) and read-only memory (ROM), input/output (IO) interfaces 1908a, 1908b 
& 1908c, a video interface 1910, and one or more storage devices .generally represented 
5 by a block 1912 in Fig, 19. The storage device(s) 1912 can consist of one or more of the 
following: a floppy disc, a hard disc drive, a magneto-optical disc drive, CD-ROM, 
magnetic tape or any other of a number of non-volatile storage devices well known to 
those skilled in the art. Each of the components 1904 to 1912 is typically connected to 
one or more of the other devices via a bus 1914 that in turn can consist of data, address, 
10 and control buses, 

The video interface 1910 is connected to the video display 1916 and provides video 
signals from the computer 1902 for display on the video display 1916. User input to 
operate the computer 1902 can be provided by one or more inptt devices 1908b. For 
example, an operator can use the keyboard 1918 and/or a pointing device such as the 
15 mouse 1920 to provide input to the computer 1902. 

The system 1900 is simply provided for illustrative purposes and other 
configurations can be employed without departing from the scope and spirit of the 
invention. Exemplary computers on which the embodiment can be practiced include 
IBM-PC/ ATs or compatibles, one of the Macintosh ™ family of PCs, Sun Sparcstation 
20 ™, or the like. The foregoing are merely exemplary of the types of computers with which 
the embodiments of the invention may be practiced. Typically, the processes of the 
embodiments, described hereinafter, are resident as software or a program recorded on a 
hard disk drive (generally depicted as block 1912 in Fig. 19) as the computer readable 
medium, and read and controlled using the processor 1904. Intermediate storage of the 
25 program and pixel data and any data fetched from the network may be accomplished 
using the semiconductor memory 1906, possibly in concert with the hard disk drive 1912. 

In some instances, the program may be supplied to the user encoded on a CD-ROM 
or a floppy disk (both generally depicted by block 1912), or alternatively could be read by 
the user from the network via a modem device connected to the computer, for example, 
30 Still further, the software can also be loaded into the computer system 1900 from other 
computer readable medium including magnetic tape, a ROM or integrated circuit, a 
magneto-optical disk, a radio or infra-red transmission channel between the computer and 
another device, a computer readable card such as a PCMCIA card, and the Internet and 
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Intranets including email transmissions and information recorded on websites and the 
like. The foregoing are merely exemplary of relevant computer readable mediums. Other 
computer readable mediums may be practiced without departing frorn the scope and spirit 
of the invention. 

The methods of the invention may alternatively be implemented in dedicated 
hardware such as one or more integrated circuits performing the functions or sub 
functions of methods of the invention. Such dedicated hardware may include graphic 
processors, digital signal processors, or one or more microprocessors and associated 
memories. 

13. Second Embodiment of Apparatus - Digital Video Broyvser System 

A Digital Video Browser System in accordance with a second embodiment of the 
apparatus is described in this section. The functionality of the Digital Video Browser 
System is enabled by the descriptions of digital video that are automatically generated 
using a description scheme, designed for digital video resources, such as that included in 
Appendix D. 

The Digital Video Browser System allows a user to browse the digital video in a 
non-lineax manner, manually annotate the digital video to provide additional descriptive 
information that was not able to be automatically generated, and to! search for the presence 
of various descriptors in a description. It should be clear to the reader that all this 
functionality is enabled by an interaction of the user with the description scheme and the 
individual descriptions of the digital video resources and that the browser that is described 
in the following section can in essence be applied to any other electronically-accessible 
resource. 

An example of such a Digital Video Browser System is shown in Fig. 16. The 
system contains a Video Browser Panel 3 600 which consists of a Viewing Panel 1601, a 
Table-of-Contents (or TOC) Panel 1602, and an Index Panel 1603. Outside of the Video 
Browser Panel 1 600 but within the system are three buttons required for user interaction; 
a Search button 1605, a Play button 1606, and an On/Off button 1607. 

User interaction with the panels of the Digital Video Browser System can be 
mediated by a touch-sensitive Video Browser Panel, however this feature is not necessary 
for the operation of the system. The operation of the Digital Video Browser System will 
now be discussed in the terms of Fig. 16. 
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When a new digital video resource is added to the Digital Video Browser System a 
predetermined description scheme is applied to the digital video resource resulting in the 
content creation methods of the relevant descriptor handlers in the description schemes 
being initiated. Other implementations might provide more than one description scheme 
which can be applied to the digital video resources. For exaipple, a Digital Video 
Browser System might provide the description schemes contained! in Appendices B and 
D- In such an embodiment the user would require a means to , select the description 
scheme that he/she would like to apply to each new digital video resource. So, for 
example, if he/she was adding a new digital video resource contahjdng the footage from a 
football match then he/she would most likely use the description ^cheme in Appendix B 3 
however if the digital video resource contained some footage of a recent holiday s then it's 
likely that the description scheme contained in Appendix D would be more appropriate. 

If more than one description scheme is available then the selection of the most 
appropriate description scheme to use could also be automated to some extent. The 
resource to be described could be analysed to see if it contained key features that typically 
indicate the use of a particular description scheme. For example, the sound track of a 
digital video resource could be analysed for repetitive whistle | sounds arising from a 
referee's whistle. If detected, such sounds could provide evidence for the use of a 
particular description scheme (eg., the description scheme shown in Appendix B). 

In a simple description scheme such as that included in Appendix B there is a single 
descriptor handler specified for the description (which is also a descriptors which 
generates the entire content for the description. 

I 

In other description schemes, more than one descriptor jraay have an associated 

i 

descriptor handler which is responsible for automatically generating the content of just 
that descriptor. For example,, consider the description scheme shown in Appendix D. 

i 

The VideoDescription descriptor Dl has an associated descriptor handler D2 which 
provides a method to automatically segment the digital video .resource into a series of 

individual shots. The Shot descriptor D3 has an associated descriptor handler D4 which 

i 

provides a method to automatically select a key frame from =a specific shot and then 
generate a series of semantic labels which provide some information about the content of 
the particular shot (eg., whether or not the shot contained people, was an indoors or 
outdoors shot, etc.). These descriptor handler methods are executed on the creation of a 
descriptor in the description being generated. Therefore, the description can be 
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progressively constructed using the description scheme (effectively as a template) and the 
set of descriptor handlers that provide the methods for automatically generating the 
content for their relevant descriptors. An example of such a generated description is 
provided in Appendix E, 

lii the case of the Digital Video Browser System depicted in Fjig. 16, the descriptors 
able to be accessed in the Index Panel, rather than the TOC Panel are classified as Index 
Descriptors. The classification of descriptors as Index or TOC descriptors is achieved 
using presentation rules (see Section 8, Method of Presenting Descriptions of Resources) > 
with each description scheme being used by the Digital Video Browser System having a 
corresponding presentation rule set For example, a presentation r$le could be applied to 
each of the descriptor definitions in the description scheme to add; an attribute definition 
to the descriptor's definition for the purposes of this classification. The added attribute 
definition could have a attribute default of #FIXED "Index" or #FDCED "TOC" to classify 
an Index and TOC descriptor, respectively. [Note: The use of the #FIXED keyword in 
the default value means that changing the value of the classifier, from its default value 
results in an invalid XML construct and hence an invalid description.] 

Selecting which descriptors are to be used as Index descriptors is similar to selecting 
which key words or phrases you would include in the index of a book. In other words, it 
is an authoring task that results in presentation rules. In general, a descriptor that is 
classified as a TOC descriptor represents a structural element! of the resource (ie. s a 
component that would normally appear in the TOC of a book). So, for example a Shot 
descriptor is a TOC descriptor. An Index descriptor typically represents a property of a 
TOC descriptor (eg., a Shot descriptor could contain people scenes, be an indoor or 
outdoor scene, etc). 

The Index descriptors axe the leaf nodes of the internal tree structure used to 
represent the description [The internal representation of descriptions is discussed in detail 
in Section 2.3 Description Object Model (DesOM) ]. In the absence of presentation rules, 
this property can also be used to implicitly differentiate between Index and TOC 
descriptors in an implemented Digital Video Browser System. In the Digital Video 
Browser System, explicit differentiation between Index and TOC Descriptors is achieved 
using presentation rules. A set of presentation rules applicable to the description scheme 
in Appendix D is shown in Appendix R 
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The Digital Video Browser System has access to a collection of digital video 
resources, which is hereinafter referred to a Digital Video Library. A newly described 
digital video resource can be simply appended to an existing collection of described 
digital video resources. Alternatively (see Section 11. Remote Digital Video Browser 
Devices)^ the user can insert a new item at the desired location using a drag-and drop 
means. The Digital Video Library is itself a resource able to be described. Therefore, on 
initialising the Digital Video Browser System a description scheme for a Digital Video 
Library is used to automatically generate a description for the Digital Video Library, 

The description of the Digital Video Library can be very simple containing just a 
hierarchical representation of the individual descriptions of digital video resources 
described in the library, In other words, the description need not know about the location 
of the digital video resources described in the library. It is merely a catalogue of the 
descriptions of the digital video resources stored in the library. Each individual 
description has a reference to its corresponding digital video resource. 

An example of a description scheme for a Digital Video; Library is included in 
Appendix G, The Digital Video Library's description can contain zero or more Section 
elements or zero or more Item elements, where each Item element refers to an individual 
description in the Digital Video Library (ie., an XML document). A description of a 
Digital Video Library conforming to the description scheme included in Appendix G is 
shown in Appendix tL 

During browsing the user can select sections of Digital Viideo Library by ^electing 
the relevant descriptors in the TOC Panel 1602 in the Video Brbwser Panel 1600. This 
selection method provides non-linear access to the digital video resource(s). Typically 
these selections are highlighted in the TOC panel to indicate which are currently selected. 
The user can choose to play all the highlighted selections by pressing the "Play" button 
1606. \ 

Alternatively the user can search for sections, items or parts of items of the Digital 
Video Library by selecting relevant Index descriptors in the Index Panel 1603. In a 
simple Digital Video Browser System implementation, the Index descriptors might imply 
simple boolean presence of a specified feature. For example, the PeopleScene Index 
descriptor (see D5 in Appendix D) could indicate whether people are either present or 
absent from the shot. In a more sophisticated Digital Video Browser System the Index 

i 
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descriptors might require some representative value. For exariiple, a "Date" Index 
descriptor would require a specified value before a search could be performed. 

Searches can be performed within a TOC context in the Digital Video Library, For 
example, if a user wanted to search for PeopleScene descriptors within a specific digital 
5 video resource, the user could select the TOC descriptor for that particular resource in the 
TOC Panel 1602 and then select the desired Index descriptor in the Index panel 1603 and 
press the "Search" button 1605 in the Digital Video Browser System, The search process 
would then result in all TOC descriptors that satisfied the seirch criteria becoming 
selected (eg., highlighted) in the TOC Panel 1602, The user could then select to play all 

10 the selected sections of the digital video resource by pressing the "Play" button 1 606. 

Searches can be implemented in the Digital Video Browser System using selection 
rules (see Section 10. Method of Selecting Resource Description^). The TOC context is 
automatically inserted as part of the pattern of the selection rule. The search process 
applies the selection rule pattern to each relevant description and updates a selection 

15 attribute that has been added for all selectable attributes using a presentation rule. 
Selectable attributes will vary between description scheme and application. In the case of 
the description scheme included in Appendix D the only descriptors that might be 
classified as selectable would be the VideoDescription and Shot descriptors (see the 
presentation rules in Appendix F). 

20 The Digital Video Browser System also provides functionality for manual 

annotation, in conformance with the description scheme, of a digital video resource. If a 
particular TOC descriptor is selected, then the relevant Index descriptors 1609 can be 
displayed in the Index Panel 1603. The Index descriptors are preferably represented by 
icons (which in preferably are specified by presentation rules targeted at the descriptor 

25 definitions). The selected TOC descriptor can be viewed (played) and then manually 
annotated by dragging icons representing the Index descriptors (eg., 1610) into an 
Annotation Region 1604 of the Viewing Panel 1601. Annotations created in this fashion 
are then added to the description of the resource and are available for subsequent 
browsing. 

30 Annotations in the form of titling various TOC Descriptors could also be possible in 

some implementations of a Digital Video Browser System. For example, in a Digital 
Video Browser System implemented in software on a regular persona] computer, the 
screen representation of the Descriptor could be selected afrd then the title for the 
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descriptor could be entered using the computer's keyboard. In alternative embodiments, 
" in which access to the Video Browser Panel 1600 is provided !via a touch-sensitive 
display, user entry of textual titles could be mediated by a pen interface or via a method 
whereby a particular descriptor is selected by touch, and the title 'communicated by the 
5 user speaking the title words and a speech-to-text module in the Video Browser System 
converting the spoken words to text and displaying the result where a title is expected on 
the display. I 

Whenever new descriptions are retrieved for browsing the description is processed 
into a DesOM, Before the description is actually presented in the Video Browser System, 

10 any inference or equivalence rules (see Section 8. Method of Extending Descriptions of 
Resources) that are associated with the description's description scheme are processed. 
This processing involves iterating through the defined inferencje rules until no more 
changes can be made to the description. Clearly, this rule processing requires that there 
are no circular dependencies in the rule set. The inference and/ctf equivalence rules will 

15 result in the creation of new descriptors which have been inferred from those that were 
part of the serialised description. Preferably, any new descriptors; created by this process 
will have been defined as part of the relevant description scheme (and as such will have 
been classified as an Index or TOC descriptor). The inference rules will need to be 
reprocessed in the event of any annotations being created. j 

20 14. Third and Fourth Embodiment of Apparatus - Remote IMgital Video Browser 
Devices 

The Digital Video Browser System described in the previous section can also be 
implemented as a dedicated remote device. In this section two possible remote device 
embodiments of the Digital Video Browser System are described with respect to Fig. 17 
25 and Fig. 18. 

The first remote device of the Digital Video Browser System is shown in Fig, 17. hi 
this embodiment the Video Browser 1700, contains no storage for the Digital Video 
Library. The Video Browser 1700 communicates with a Serv6r 1710 using a wireless 
transmitter/receiver 1702 and a wireless connection 1703. The Server 1710 has a 
30 connection 1717 with a storage device that contains the Digital Video Library 171 L All 
the digital video resources that can be browsed by the Video Browser are stored in this 
Digital Video Library. Preferably, in this remote device all the descriptions of the digital 
video resottrces are also stored in this library 1711. The 'Server 1710 also has a 
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connection 1714 to a large display 1712 that can be used for public viewing of the digital 
video resources. Preferably, the connections between the Server 1710 and the Digital 
Video Library 1711 and between the Server 1710 and the large display 1712 are wired 
connections. 

New digital video resources can be added to the Digital Video Library 1711 which 
is directly connected to the Server 1710 independently of the Video Browser device 1700. 
As the resources are added to the Digital Video Library 1711 (from; for example, a digital 
video camera), descriptions for the digital video resources are automatically generated 
using the description scheme. Also at this time, usually after the description has been 
generated, the user could optionally title sections of the digital video resource. These 
titles would then be visible when browsing using the Digital Video Browser device. 

On power-up the Video Browser device connects to the Server 1710 using the 
wireless connection 1703. The Server 1710 communicates to the Digital Video Browser 
device a description of the Digital Video Library, This description, like descriptions of 
the digital video resources, conforms to a description scheme (in; this case for a Digital 
Video Library), and is serialised in an XML document. An example of a description of a 
Digital Video Library is shown in Appendix H. 

The remote Digital Video Browser device 1700 can either store the relevant 
description schemes permanently, or download these description ;schemes at the time of 
making its connection with the Server 1710. The latter method of obtaining the 
description schemes is preferred. The description of the Digital! Video Library .and the 
relevant description schemes contain all the information required jto display an Index and 
TOC panel on the Digital Video Browser device 1700, The user can then use the Digital 
Video Browser device to navigate through the Digital Vide<j> Library, selecting or 
searching for video resources to view. Preferably, the navigation through the TOC and 
Index panels is enabled via a touch-sensitive screen. Other methods of navigation (eg,, a 
pen or simple keyboard) could also be used- 
Only when a Digital Video Browser user selects to "Play" : a particular selection of 
digital video resources, is ii necessary to transmit the required^ digital video resources 
from the Digital Video Library 1711 to the remote Digital Video Browser device 1700. 
Preferably the digital video resources are stored and transmitted in compressed form (eg., 
MPEG-1 or MPEG-2 ) therefore minimising the bandwidth of the required wireless 
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connection 1703 between the Server 1710 and the remote Digital Video Browser device 
' 1700. j 

The remote Digital Video Browser device can optionally have?' an additional button 
(to those shown in Fig. 16), which can be used to direct the Viewing Panel 1701 of the 
5 remote Digital Video Browser device to a large display 1712 connected to the Server 
1710. This redirection can be achieved by transmitting a description of the required 
presentation (ie., an XML document) from the remote Digital Videol Browser device 1700 
to the Server 1710. This description would conform to a Video Presentation Description 
Scheme (eg Appendix I) that could be as simple as just a list of all the selected sections of 
10 the selected digital video resources. An example description of a! video presentation is 

shown in Appendix J. I 

i 

The video browser system generates a description of a videp presentation by first 
reading the description scheme for the presentation (eg Appendix I). This description 
scheme contains definitions of descriptors required for the video presentation (eg 

15 VideoDescription Reference, Shot Reference descriptor definition^ in Appendix I)* The 
video browser system then generates the description of the presentation using the 
description scheme read by the browser and information about the resources that have 
been selected for presentation. The result of this step is a description such as shown by 
way of example in Appendix I. This description can then be directed to an output device 

20 1712 for rendering. I 

Preferably, this description is interpreted by the Server 171© and the corresponding 
sections of the selected digital video resources would be rendered to the large display 
1712. The rendering is perforated by the Server 1710 and pixel d^ta would be transmitted 

over the connection 1714, however if the large display 1712 had [the processing ability to 

i 

25 decode the compressed digital video resource, then the comprised resource could be 
transmitted over the connection 1714 and then decoded and rendered in the large display 
1712. 

Clearly, presentation rules could be applied to the presentation of the selected items 
in the same way as presentation rules are applied to a description of a digital video 
30 resource- Some presentation rules that could be applicable to the presentation of digital 
video resources include rules that specify the type of transitions to be inserted between 
shots of a particular digital video clip (eg., fades, cuts, wipes, etc.) and whether clip titles 
are to be rendered over the presented video and the style of title rendering to be used. 
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These rules could be collected in a presentation rule set that is linked with the Video 
- Presentation Description Scheme in the same way that sets of presentation rules could be 
linked to the Digital Video Resource Description Scheme (see Appendix D). 

Alternative Digital Video Browser implementations could allow users to specify 
5 additional presentation rules for the presentation of selected digital video resources. For 
example, an implementation could allow a user to specify whether a particular selection 
was to be played at recorded, slow or fast speed. Altering the spe^d of video playing can 
provide interesting presentation effects. Similarly, the Digital Vidjeo Browser user might 
also be able to specify the types of transitions to use on a one-off presentation basis rather 
10 than a default basis as provided by rules linked to the Video Presentation Description 
Scheme. These one-off presentation rules can be combined into a [single rule set which is 
referenced by the Description element of the presentation description that is 
communicated to the Server 1710 when the user chooses to play the selected digital video 
resources (whether on the Digital Video Browser device itself or' more likely, when the 
15 presentation has been re-directed to the large display 1712). 

An example of a Video Presentation Description Scheme, which could be used with 
the Video Description Scheme shown in Appendix D, is shown; in Appendix 1 In this 

description scheme, a standard set of presentation rules is provided as part of the 

j 

description scheme. These rules have been collected into a rule sdt and stored in the XML 
20 document which, in the case of the example is called "VideoPresdntationRules.xmJ". The 
rule set has then been referenced by the description scheme by specifying an ENTITY for 
the ruleSets attribute II of the VideoPresentationDescription element. The attribute 
userPresentationRules 12 has been added to the VideoPresentationDescription subclass of 
the Description element to be able to contain an ENTITY that specifies an xml document 
25 that contains any presentation-specific rules. J 

An example of a video presentation description that conforms to the Video 
Presentation Description Schema which is included in Appendix I, is shown in Appendix 
J. A set of presentation-specific rules has been specified for the particular presentation 
using the userPresentationRules attribute of the VjdeoPresentationDescripiion element 
30 (see 11). Clearly the example description scheme and presentation description included in 
Appendices I and J pertain to the Video Description Scheme included in Appendix D 
since they refer to particular descriptors m that description scheme. For example, the 
VideoDescriptionReference element contains zero or more references to Shot elements in 
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the referenced video descriptions. In particular the shotlDRef element J2 specifies a 
particular shot descriptor in the description contained VideoEgLxrnl; by using a reference 
to the ID of that descriptor in the description. It is not necessary to use a Video 
Presentation Description Scheme that is directed so specifically at a particular description 
scheme. For example, if a Digital Video Browser System was implemented with more 
than one description scheme, then a more general Video Presentation Description Scheme 
can be used. 

The ability to be able to re-direct the Viewing Panel 1701 16 a large display 1712 
connected to the Server 1710 is a useful feature as the user canlselect the sections of 
his/her Digital Video Library that he/she wishes to share with an audience using the 
remote Digital Video Browser device. That selection can then sipiply be played to the 
large display 1712. 

A second remote device implementation of the Digital Vidieo Browser System is 
shown in Fig. 18. In this implementation the Digital Video Browser 1 800 is implemented 
as a remote device that has a capability to read Digital Video Disks (DVDs). Typically 
each DVD is treated like an independent Digital Video Library £nd consequently each 
DVD has its own description of the Digital Video Library contained on the DVD. When 
the DVD 1815 is inserted into the remote Digital Video Browser device 1800 the Video 
Browser 1800 reads the description of the Digital Video Library contained on the DVD. 
In this device the description scheme required to interpret the Digital Video Library would 
preferably reside in the remote Digital Video Browser device, hdwever it is conceivable 
that the description scheme could also be located on each DVD. Similarly the description 
schemes required to interpret the descriptions of the digital video resources could either 
be located on the DVD or in the remote Digital Video Browser device. In the preferred 
implementation of this device, all the required description schemes are located in the 
remote Digital Video Browser device 1 800. New description sihemes for digital video 
resources can be downloaded via the wireless transmitter/recqiver 1802 and wireless 
connection 1804 to a server or computer 1813 connected to a network 1814. 
Alternatively, the remote Digital Video Browser can be docked at a server or networked 
computer for the download of new description schemes. 

Once the description of a Digital Video Library has been read from the DVD IS 15 
then the user can navigate through this Digital Video Library as described previously. 
Sections of described digital video resources can be selected a^d played on the remote 
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device. The device is in many respects very similar to the device depicted in Fig. 17, with 
the exception that it does not require a Server to store the digital video resources and 
descriptions. 

Sections of the selected digital video resources can be selected for viewing on a 

i 

5 large display 1810 that has a wireless connection 1803 with the: remote Digital Video 
Browser device. This large display 1810 must either contain, or be directly connected to, 

a processor able to decode and render the compressed digital video resource that is 

i 

transmitted via the wireless connection 1803. As with the remote idevice depicted in Fig, 
17, a description of the required presentation is communicated to ;the large display 1810. 

10 In addition, any digital video resources required for the presentation description to be 
rendered must also be communicated. These resources are typically communicated in 
compressed (encoded) form (eg., MPEG-1 or MPEG-2), The processor either contained 
in, or directly connected to, the large display 1810 renders the presentation using the 
presentation description and its associated digital video resources; The rendering process 

15 can typically adapt to the resolution of the large display 1810, which is usually greater 
than that of the handheld device. • 

In the preferred implementation of this device 1800, if the description of the 
required presentation requires that only particular sections of & selected digital video 
resources be presented, then these required sections can be isolated from the original 

20 digital video resource, recorded if necessary in the handheld device, and then 
communicated to the large display 1810. This approach reduces the communication 
bandwidth of the wireless connection 1803. Alternatively, the entire digital video resource 
can be communicated and the processor that renders the presentation will need to extract 
the relevant sections of the digital video resource(s). The latter implementation is more 

25 costly in bandwidth but does not involve recording of digital video resources in the 
remote device. 

In order to facilitate resource discovery on different DVDs, this remote Digital 
Video Browser device can also have an ability to generate printed DVD covers that 
display the contents of the DVD in a graphically pleasing manner. This printed 
30 presentation can be achieved in substantially the same maimer as described for a vide 
presentation. 

This facility can be achieved using a wireless connection 1805 to either a printer with 
some processing ability 1812, or to a computer directly connected to a printer (not shown 
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in Fig. 18). Typically the Digital Video Browser device would send to the printing device 

(1812 or the computer directly connected to a printer), a description of the (printed) 

i 

presentation that is to be the printed DVD cover, j 

Description schemes for this presentation could be designed just as they can be 
5 designed for video presentations (eg., see Appendix I), For example, at the simplest level 
the Digital Video Library description could fomi the basis of the printed presentation. 
Presentation rules could then be used to specify the spatial layout and colour arrangement 
of the printed presentation, and also the association of icons or key frames to particular 
descriptors in the description. The presence of visual reminders of the content of the 

10 DVD, such as icons or key frames, are important for purposes of identification and 
retrieval. ; 

A processor, which is located either in the printer 1812 or in a computer connected 
to the printer could then use the description of the required printed presentation and any 
provided presentation rule sets to render a DVD cover for the particular DVD using the 

15 provided key frames. This processor would need to be able to interpret the description of 
the printed presentation. ; 
15. Fifth Embodiment of Apparatus - Media Browser System 

Browsing of electronically-accessible resources other than digital video can also be 
enabled by descriptions that conform to identified description schemes. In an alternative 

20 embodiment, a Media Browser System can enable the description-based browsing of any 
electronically-accessible resource. Although the description schemes used to describe 
these different resources might be significantly different, a commbn browsing framework, 
called here a Media Browsing System, can be used. The Video Browser System described 
in Section 13 is a more specific embodiment of the Media Browser System described in 

25 tiiis Section. However, many aspects of the Video Browser System can also be 
implemented in the Media Browser System. I 

The browsing method requires that each resource is consistently described (eg., using 
the DDF) according to a description scheme and the resulting description contains a link 
to the resource or sections of the resource. Preferably, die DDF (see Section 2) is used lo 

30 provide a consistent method of describing resources, however alternative methods of 
describing resources could also be used. For example, other schema languages such as 
XML-Schema of the W3C could also be used. In the case of XML-Schema, core 
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descriptor elements can be defined in substantially the same way ds described for the DDF 
in section 3.1.2, j 

In addition, in the embodiment described here, descriptor ccjmponents of description 
schemes are further classified using predetermined classifications that provide axes of 
5 access to the resources. The preferred axes of access used in this embodiment are the 
structural access (Table of Contents(TOC) access) and the index access. These axes have 
been used because humans are familiar with their use in, for example, reference books. 
Whereas the TOC-axis of access provides access to resources onjthe basis of context (ie., 
where a resource or section of a resource exists in relation to otifier resources or sections 
10 of resources), the index-axis effectively provides context-free access to resources Gust as 
an index in a reference book). It should be clear to those skilled in the art of browsing 
technologies that the value of this classification of descriptor components into TOC and 
index axes of access is that a Media Browser System can act both as a browser (in the 
sense of cuiTent web-browsing technology) and a search engine iii one. 

15 It is possible to use different axes of access and the number! of axes is not limited to 

i 

two. For example, the Media Browser System could use an interface similar to that shown 
in Fig. 16 for the Digital Video Browser System, but having a further axis of access on the 

left hand side of the viewing panel to provide access to the digital! video via audio events. 

j 

Another variation is one where more than one TOC axis can be lused to allow more than 

j 

20 one structural view to browsable content- For example, one TOCj might provide browsing 
access using content category (eg. birthday images) and another apcis might provide access 
by date of creation. 

The predetermined classification of descriptor components into axes of access can be 
achieved using the methods described for the Digital Video Systeim (see Section 13). 

25 Any electronically-accessible resource can be accessed by thje Media Browser System 

using a description of the resource as long as the Media Browser System can access the 
description scheme to which the description conforms and a processor for the class of 
resource. For example, a digital video resource might require a MPEG-1 or MPEG-2 
processor (player) to be present, an image might require a JPEG viewer and an audio 

30 object might require an MP3 audio processor. These processors are preferably stored with 
the browser and new processors are able to be downloaded wheh required by a resource. 
The Media Browser using the DDF, identifies the processor required by the resource by 
the relevant NOTATION declaration in the description scheme (see Section 3. 1.2.2), 
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The resources can be an electronic document or other resources available over the 

i 

web. The resource can also be an electronic device. The description that appears in the 
TOC axis can also be located at different sites on the web- In thisisense, the TOC axis can 
be compared to a set of description bookmarks. A TOC item m^y contain links to other 
5 descriptions, to individual resources or sections of resources (eg. a spatio temporal extent 
in a digital video). 

It should be clear to someone skilled in the art that if resource library providers on 

the web described their resources using a consistent method such as the DDF, a TOC axis 

i 

could be made to extend over all resource libraries of interest to a particular user. In other 

10 words, the TOC could represent an information landscape over which a user could browse 
and search for resources. This has the advantage of the user not having to visit each digital 
resource library site in turn in order to search for desired resources. 

The media can be browsed, annotated and searched in the same way as that described 
for digital video resources (see Section 13). Clearly, the descriptors that appear in the 

15 different axes during browsing will vary depending on the description schemes that are 
relevant with regard to the browsing context at any particular instance. For example, if 
more than one description is currently in context and these (descriptions conform to 
different description schemes then an index panel will reflect all the descriptor 
components which have been classified as index descriptors in; the relevant description 

20 schemes. In other words, the set of index descriptors that are shown at any time in the 
Media Browsing System represents the union of the sets of in4ex descriptors fyat arise 
from all the description schemes that are relevant to the descriptions that are currently in 
context. In other embodiments, it would be possible to show onljy those index descriptors 
that represent the intersection of the sets of index descriptors that arise from the relevant 

25 description schemes. In this case, an index descriptor would need to exist in each of the 
relevant description schemes before it could be provided by the sMedia Browsing System 
as an index for browsing. : 

The selection of a TOC context for searching using the incjex panel is implemented 
in the Media Browser System using the method that is describled for the Digital Video 

30 Browser System (see Section 1 3) 

Links between descriptor components of descriptions and spatially and/or temporally 
localised sections of the resources can be represented in the descriptions using locators 
and extents (see Section 3.1 A for how these constructs are used in the context of the 
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DDF). Preferably, the navigation of these links is performed automatically by the Media 
Browser when the user selects to play/view the selected resource(is). The Media Browser 
identifies the spatial and/or temporal extent(s) that are linked to d selected descriptor and 
plays/views the section(s) of the resource which contain the extends). 
5 For example, a descriptor could refer to a spatially and temporally localised object in 

a digital video. This object can be represented spatially by a rectangular bounding box 
and temporally by a range of image frames. If a user selected the object's descriptor and 
chose to play the selections, the Media Browser would automatically identify the frames 
which contained the object and play that section of the video only.! 

10 The Media Browsing System differs from existing HTML browsers in that the 

entities mediating the browsing [eg., the (DDF) descriptions] contain only descriptions of 
the resources to be browsed, in the case of the HTML browseij, the HTML documents 
represent the resource, control the presentation of the resource^ and also contain some 
description of the resource (the META tag). The browser typically does not use the 

15 descriptive information. In contrast, the entities mediating the 'browsing in the Media 
Browser System are ONLY descriptions of resources. These descriptions contain links to 
the relevant resources or sections of the resource to be viewed or played. The key and 
most obvious advantage obtained by browsing using descriptions of resources is browsing 
access to non-textual resources (eg., digital signals) However, the Media Browser style 

20 of browsing also uses the descriptive information available about resources to provide a 
richer browser interface that can include annotation and searching. This richer inlprface is 
also available to textual documents (eg., XML and HTML documents). 

Fig. 20 shows an example of the Media Browsing System in accordance with the 
fifth embodiment. The Media Browser system 2000 contains £ viewing pane 2006, a 

25 Table of Contents ( or TOC) panel 2002 and an Index panel 2004 The TOC panel 2002 
displays a Table of Contents of all resources and resource libraries that are of interest to a 
user. The TOC effectively represents a personal library where individual items are 
typically distributed across the web. It is like a set of bookmarks or pointers into different 
description libraries or single descriptions. 

30 The TOC axis provides an information landscape for the user to browse. Individual 

items of TOC can be: 

(i) expanded to show contained items or collapsed to hide cqntaincd items; 

(ii) selected to play/view; j 
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(iii) selected for the current context (eg. to search). 

Preferably, these browsing functions are enabled in the following way. Each TOC 
item consists of a node symbol (eg. a large bullet symbol) and a bode content ( which is 
defined by the corresponding descriptor in the description). The node content can be an 
5 image ( such as a key frame as is preferably used to represent a section of video) or some 
text which can describe the personal item ( whether it represents a section in the personal 
library or information landscape, or a resource), ! 

TOC items are expanded and collapsed by checking on the node symbol (ie. the node 

i 

symbol acts as a toggle). It is preferable for the node symbol to indicate whether an item 
10 can be expanded. For example, the node symbol can be displayed as an open bullet 
symbol if it has contained items (ie. can be expanded) and a filled bullet symbol if it 
cannot b e expanded) , 

TOC items can be selected for viewing/playing by clickiiig on the node content. 
Preferably, a single click indicates that an item should be queued to be viewed/played. A 

15 double click action results in immediate viewing/playing of an item. Items that are queued 
for viewing/playing are only played when the user selects to present the media. A Media 
Presenter tool is described later in this Section. Preferably, a button appears in the control 
region 2016 to initiate presentation. Pressing this button involves the current presentation 
tool which is a plug-in tool of the user's preference. It should be clear that many such 

20 plug-in tools could be used. If a user was only interested in images then only an image 
plug-in tool would be required. If a user was browsing a rangej of content then a more 
sophisticated plug-in tool would be required. This tool would need to be able invoke more 
specific tools for the playing/viewing of different type of resources. 

Preferably, items that have been selected are differentiated from unselected items by 

25 highlighting the node content ( eg displaying a node content's teict in bold or highlighting 

i 

the frame of an image or visual icon). Selection of an item that contains other items 
automatically selects the contained items. ; 

TOC items can also be selected for context. Preferably, context selection is achieved 
by right clicking the node symbol. This action results in a coloured frame being displayed 
30 around both node symbol and the node content. Any context can be removed simply by 
right-clicking on the node symbol (ie. right-clicking on the node symbol acts as a toggle 
for select for context, just as left click on the node symbol acts as a toggle for 
expand/co) lapse. Right clicking of the node content of an itenji can be used to display 
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properties of the corresponding item. Preferably, these properties contain the index 
descriptors that pertain to that node item. - 

It should be clear to those skilled in the art that the browsing functions of the TOC 
axis can be implemented in many different ways without departing from the spirit and 
5 scope of the invention. 

Preferably each axis of the Media Browser System can be scrolled. This means that a 
section of the TOC can be retained in an expanded form and a further section scrolled into 
view. The properties can be displayed in a small panel (like a callout) adjacent to the node 
content. 

10 Initially, the Index panel 2004 displays all index items associated with all the table of 

i 

contents items. In the present example, it can be seen that the item "Images - Birthdays 7 ' 
2008 of the Table of Contents has been currently selected for context by the user. The 
Media Browser System 2000 then displays in the index panelj 2004 a list of indices 
determined by the description schemes that correspond to the cu|rrent]y selected item of 

15 the Table of Contents (eg, in this case birthday images). 

The Media Browser System also allows the user to further describe or annotate TOC 
items. Annotation can be achieved by allowing the user to drag; a displayed index item 
onto displayed TOC items. An annotation of this form is only; allowed if the dragged 
index item is a valid descriptor for the corresponding TOC descriptor. For example, a user 

20 would be allowed to drag a ( Teople" index item onto a particular birthday image if the 
corresponding "People" descriptor was a valid descriptor for the image descriptor. The 
"People" descriptor is an example of a descriptor not having a representative value, ie. it 
acts like a boolean indicator of the presence of a person in, in this case, an image. Many 
index descriptors require representative values to be specified ajs part of the annotation 

25 process. In these cases, as the required index item is dragged onto the TOC item, if the 
corresponding descriptor is allowed for that TOC item, a field or edit box is displayed for 
the user to enter the required representative value. Preferably, the Media Browser System 
ensures that the entered representative value is in the required fopn for the descriptor eg. 
dates may be specified to conform to a particular ISO standard. Datatyping of 

30 representative values is discussed in Section 2. 

Index descriptors that have been added to TOC items can be viewed by selecting to 
view the properties of the TOC items. Preferably, properties, of TOC items can be 
displayed by right clicking the node content. j 
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Preferably, all descriptors are viewed as being annotable. In the event a description's 
origin is a remote database which is not available for update, the annotations are stored 
locally as a copy of the updated description having a link to the remote description. The 
link for the relevant TOC item is updated to point to the updated local copy of the 
5 description. In a subsequent browsing session, the local copy of the description is read and 
the description which is used by the browser is constructed by obtaining the remote 

description and then modifying this description according to the iocal copy. This method 

i 

ensures that changes in tbe remote description are available to thejuser ( ie. the local copy 
does not simply overwrite the original description). An alternative! way of achieving this is 

10 to only store locally annotations in a partial description form. The form of these 
annotations could be defined by a special description scheme. 

Another variation of the annotation procedure would be to allow read-only 
descriptors. For example, the core description element (as deflneji in Section 2) could be 
amended to include a read-only attribute. If a description was classified as a read-only 

15 item then a user would not be permitted to annotate TOC items corresponding to that 
dcscription. 

Preferably, the index panel 2004 contains an input box 2Q14 associated with each 
index item for user entry of a query. In this way, a user may for example enter a date 
query ( eg. July 1999) in the input box associated with date incjex. The Media Browser 
20 will then highlight in the TOC any TOC items, that satisfy the query (ie. have a date value 
of July 1999) and are contained in the currently selected context pf the TOC (ie,. birthday 
images). 

Preferably, the index panel 2004 also contains a input box 2010 for a user entry of a 
free query. This input box 2010 is used as the input interface for & searching engine across 

25 all description schemes. Preferably, the free query is entered $s natural language then 

I 

subsequent processed into a structured query which uses index discripors that correspond 

j 

to the TOC context. j 

Alternatively, searching functionality can be provided by a plug-in tool that uses the 
Media presentation pane 2006 to help the user construct a query using the index panel 
30 2004. This plug-in tool can be invoked by a user pressing a search button that can be 
located in the control region 2016 at the bottom of the screen. The search tool can allow a 
user to construct a query by dragging index items ( which correspond to index descriptors) 
from the index panel 2004 to the Media presentation pane 2006. jrhe plug-in tool can also 
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allow a user to combine various descriptors using the logical connectors typically used 
with search engines (eg. AND, OR, NOT etc), and allow a user to formulate a free text 
query. Clearly any free text queries, whether entered using the seajrch plug-in tool or the 
input box 2010 would need to formulated in terms of descriptor cpmponents. Inferencing 
techniques ( as employed in some expert systems) can be used for this purpose. The 
separate search plug-in tool could also optionally display the results of the search in the 
Media presentation pane 2006 and allow the user to select and play particular items 
returned by the search. 1 

Other plug-in tools can provide additional functions to be applied to selected content. 
Each of these tools could be invoked in the manner described above for the search tooL 
Alternatively, these tools could be invoked by a pull down menu option. These additional 
functions could include emailing selected items to selected peopje (using, for example, 
addresses from an address book of a commonly used email tool), or generating an 
automatic presentation based on the selected content. The latier example could use 

stylised templates that make presentation decisions based on the descriptor components of 

i 

selected resources. 

The foregoing description of the Media Browser System assumes that descriptors 
are classified according to their axis of access either by attributes which are part of the 
description schemes or by using a set of rules for adding the attributes to the description 
schemes. In the latter case the rules may be associated only with the Media Browser 
application or can be more widely used in, for example, other applications. 

In the event that the classifications cannot be achieved with one of the foregoing 
methods, axes of access classifications for individual descriptors can be inferred by the 
Media Browser System. This inferred classification can use information about the base 
elements defined using the DDF. For example, a descriptor could be classified as a TOC 
descriptor if it is directly associated with either a resource or a section of the resource. If 
descriptions are generated according to the DDF then any; descriptor which is a 
specialisation of a description element will have an associated resource through its 
definition (see Section 3.1.2.2) and any descriptor that contains a linking element to a 
section of a resource (see Section 3.1.4) will have an associated resource through the 
target of it link, These descriptors could be inferred to be TOC descriptors. All remaining 
descriptors could be treated as index descriptors. Although this method of classifying 
descriptors might not be ideal (eg., all non-TOC descriptors jmight not appear to be 
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sensible index descriptors), it does enable the Media Browser System to present 
descriptions not having sufficient presentation rules. 

The foregoing only describes a small number of embodiments of the present 
invention, however, modifications and/or changes can be made thereto by a person skilled 
5 in the art without departing from the scope and spirit of the invention. The present 
embodiments are, therefore, to be considered in all respects to|be illustrative and not 
restrictive. 1 
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Appendix A: Core DDF Element Definitions 

Corcddf 



<!- 



Core.ddf: Contains the definitions of core DDL elements 



<!NOTATION JavaClass SYSTEM "java"> 

<!ENTITY % DataTypes "(Int | Float | Double | String | Date | Time | ID | IDREF 
EDREFS | ENTITY | ENTITIES)'^ 

<! J 

Definition of core elements 



<! ELEMENT Descriptor (ANY)> 
<!ATTLIST Descriptor 
id 

xmLlang 
dataType 
superElement 
handler 



ID 

CDATA 
%DataTypes; 
NMTOKEN 
ENTITY 



<! ELEMENT Description (Descriptor^ 

<! ATTLIST Description 

superElement NMTOKEN 
resource ENTITY 
dateResourceLastModified CDATA 
ruleSets ENTITIES 



#IMPLIED 

"en" 

"String" 

#1MPLIED 

i 

#IMPLIED 



#FTXED "Descriptor' 
^REQUIRED 
^IMPLIED 
#IMPLIED 
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<! ELEMENT ParallelSequence (Descriptor+)> 
<!ATTLIST ParallelSequence 

superElement NMTOKEN 

> 

<!ELEMENT SerialSequenee (Descriptor+)> 
<!ATTLIST SerialSequenee 

superElement NMTOKEN 

> 

<! ELEMENT Neighbours (#PCDATA)> 

<! ATTLIST Neighbours 

superElement NMTOKEN 
dataType %DataTypes; 

> 

<! ELEMENT Before (#PCDATA)> 

<! ATTLIST Before 

superElement NMTOKEN 
dataType %DataTypes; 

> 

<!ELEMENT After (#PCDATA)> 

<! ATTLIST After 

superElement NMTOKEN 
dataType %DataTypes; 



#FIXED "Descriptor" 



#FIXED "Descriptor" 



#FIXED "Descriptor" 
#FIXED "IDREFS" 



#FIXED "Descriptor" 
#FIXED "IDREFS" 



#FIXED "Descriptor" 
#F1XED "IDREFS" 



I 
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<!ELEMENT InFrontOf (#PCDATA)> 

<! ATTLIST InFrontOf 

superElement NMTOKEN 
dataType %DataTypes; 

> 

<! ELEMENT Behind (#PCDATA)> 

<! ATTLIST Behind 

superElement NMTOKEN 
dataType %DataType$; 



<iELEMENT CLink (#PCDATA)> 

<! ATTLIST CLink 

superElement NMTOKEN 
dataType %DataTypes; 

> 

<! ELEMENT ELmk (#PCDATA)> 

<! ATTLIST ILink 

superElement NMTOKEN 
dataType %DataTypes; 



#FIXED "Descriptor" 
#FIXED "IDREFS" 



#FIXED '^Descriptor" 
#F3XED "IDREFS" 




#FIXED "Descriptor" 

#F1XED "IDREF" 

i 



#FIXED "Descriptor" 
#FIXED "IDREFS" - 
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<!ELEMENT Locator (Extent+)> 

<!ATTLIST Locator 

superElement NMTOKEN 
resource ENTITY 

>* 

<tELEMENT Extent (Descriptor+)> 
<!ATTLIST Extent 

superElement NMTOKEN 

> 

<! ELEMENT ImageExtent (Descriptori-)> 
<!ATTLIST ImageExtent 

superElement NMTOKEN 



#FTXED "Descriptor" 
#REQUIRED 



#FDtED "Descriptor" 



#FDCED "Extent" 



#FrXED "ImageExtent" 



<! ELEMENT RectlmageExtent (RectlmageExtentXO, RectlmageExtentYO, 

t 

RectlmageExtentHeight, RectImageExtentWidth)> 
< ! ATTLIST RectlmageExtent 

superElement NMTOKEN 

> 

<! ELEMENT RectlmageExtentXO (#PCDATA)> 
<! ATTLIST RectlmageExtentXO 

superElement NMTOKEN 

dataType %DataTypes; 

> 

<! ELEMENT RectlmageExtentYO (#PCDATA)> 
<! ATTLIST RectlmageExtentYO 

superElement NMTOKEN 

dataType %DataTypes; 



#FIXED "Descriptor" 
#FIXED "Inf 



#FIXED "Descriptor' 
#FIXED "Inf 
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<! ELEMENT RectlmageExtentHeight (#PCDATA)> 

<! ATTLIST RectlmageExtentHeight 

superEIement NMTOKEN 
dataType %DataTypes; 

> 

<! ELEMENT RectlmageExtentWidth (#PCDATA)> 

<i ATTLIST RectlmageExtentWidth 

superEIement NMTOKEN 
dataType %DataTypes; 



#FDCED "Descriptor" 
#FIXED "Lit" 



#fclXED "Descriptor" 
#FDCED "Int" 



#FDCED "Extent" 



<\ ELEMENT VideoExtent (VideoExtentStart, VideoExtentEjid, ImageExtent?)> 
<!ATTLIST VideoExtent i 
superEIement NMTOKEN 

> 

<!ELEMENT VideoExtentStart (#PCDATA)> 
<! ATTLIST VideoExtentStart 

superEIement NMTOKEN 

dataType %DataTypes; 

> 

<! ELEMENT VideoExtentEnd (#PCDATA)> 
<! ATTLIST VideoExtentEnd 

superEIement NMTOKEN 

dataType %DataTypes; 

> 



#FIXED "Descriptor" 
#FIXED "Int" 



i^FIXED "Descriptor" 
&FIXED "Int" 
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Appendix B: An Example Description Scheme for an Australian 

Football League Game 

i 

Description Scheme (AFLGame.ddf) I 



10 



15 



20 



25 



30 



<!-- Core.ddf included here — > 
<!ENTTTY % Core SYSTEM "Core.ddf >> 
%Core;^ 

B2 

<! 



Scheme specific entities 



<! ENTITY AFLGameGen SYSTEM "AFLGameGen. class" NDATA 

JAVACLASS> 
<! ENTITY % PlayType "(Mark [ Kick | Handball | Tackle) 77 >j 



<!- 



Element definitions 



B3 



<!ELEMENT AFLGameDescription (Game, Locator*)> 
<IATTLIST AFLGameDescription 

i 

superElement NMTOKEN #FIXED "Description" 



handler- 



ENTITY 



#FIXED "AFLGameGen" 



> B4 

i 

<!ELEMENT Game (Location, Date, TeamName*, Quarter*)> 
<!ATTLIST Game 

superElement NMTOKEN #FIXED "Descriptor" 

i 

> : 
<! ELEMENT Location (#PCDATA)> ! 
<! ATTLIST Location 

superElement NMTOKEN #FIXED "Descriptor" 

<!ELEMENT Date (#PCDATA)> ; 
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<! ATTLIST Date 

superElement NMTOKEN 
dataType %DataTypes; 

> 


#FEXED "Descriptor" 
#FIXED "Daie" 


5 


<!ELEMENT TeaniName (#PCDATA)> 
<! ATTLIST TeamName 

superElement NMTOKEN 

> 


#FIXED "Descriptor" 


10 


<! ELEMENT Quarter (Play*)> 
<!ATTLIST Quarter 

superElement NMTOKEN 


#FDOBD "Descriptor" 

f 


15 


<! ELEMENT Play (PlayerNo, PlayType, Annotator, CLink*)> 
<!ATTLIST Play 

superElement NMTOKEN #FDCED "Descriptor" 


20 


<! ELEMENT PlayerNo <#PCDATA)> 

<! ATTLIST PlayerNo 

superElement NMTOKEN 
dataType %DataTypes; 

> 


1 

#FDCED "Descriptor" 

#FDCED "Int" 

i 


25 


<!ELEMENT PlayType (EMPTY)> 

<! ATTLIST PlayType 

superElement NMTOKEN 
value %PlayType; 

> 


#FDCED "Descriptor" 

#REQUIRED 

i 


30 


<! ELEMENT Annotator (#PCDATA)> 
<!ATTLIST Annotator 

superElement NMTOKEN 

> 


i 

#FDCED "Descriptor" 



I 

i 
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Appendix C: An Example Description generated from the Description 

Scheme in Appendix B 

Example Description (AFLGameEg,xmi) 



<?xml version="1.0" standalone = "no" ?> 

<!DOCTYPE AFLGameDescription SYSTEM ' AFLGame.ddf' [ 

<!ENTITY MatchVideo SYSTEM "MatchVideo.mpg" NDATA MPEG2> 

]> 

< AFLGameDescription resource = tc MatchVideo"> 

<!— A description of the game is contained in this section — > 

i 

<Game> 

<!-- First some details of the game being player --> 
<Location>Sydney Cricket Ground<^Location> 1 
<Date>199S-08-09</Date> 
<TeamName>Sydaey Swans</TeamName> 
<TeamName>West Coast Eagles</TeamName>' 

<!— Now add play information with links — > 
<Quarterid = "Ql"> \ 
<Playid-"Pl"> 

<PlayerNo>23</PlayerNo> ; 
<PlayType value - "Mark"/> ; 
<Annotator>John Smith</Annotatoi> 
<CLink linkend = "Ll s 7> j 
</Play> 

<:Playid = "P2"> : 
<PlayerNo>5</PlayerNo> | 
<PlayType value «- "Kick'7> 
<Annotator>Joe Bloggs<Aanotator> 
<CLink linkend = "L2>7> 

<Play> 
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</Quarter> 


! 


* 


<Quarter id - "Q2"> . . . </Quarter> 




- 


<Quaxier id = "Q3"> . . . </Quarter> 


1 


■ 


<Quarter id = "Q4"> . . . </Quarter> 




5 


</Game> 


i 




<!— This section now contains the linkends for the varipus plays — > 




<Locator id = "LI" resource = "MatchVideo"> 


i 




<VideoExtent > 






<VideoExtentStart>0</VideoExtentStart> 




10 


<VideoExtentEnd>l 0</VideoExtentEnd> 
<RecttmageExtent> 






<RectlmageExtentX0>50<^ectImagjeExtentX0> 




<RectImageExtentY0>50</RectIma^eExtentY0> 




<RectImageExtentHeight> 1 00</RectlmageExtentHeight> 




<RectImageExtentWidth>40</Rec^ 


Ji 


</RectImageExtent> 






</VideoExtent> 






<VideoExtent> 




u 


<VideoExtentStart>l 1 </VideoExtentStart> 


Q 20 


< VideoExtentEnd>3 2</VideoExtentEnd> 
<RectImageExtent> 






<RectlmageExtentX0>80<RectImageExtentX0> 




<RectImageExtentYO>100</RectIm^geExtentYO> 




<RectImageExtentHeight>100</RectImageExtentHeight> 


25 


<RectIinageExtentWidth>40</RectIinageExtcntWidth> 




</RectTmageExtent> 






</VideoExtent> 






</Locator> 
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10 



<Locator id = "L2" resource = "MatchVideo"> ; 
<VideoExtent> 

<\ r ideoExtentStart>0<yVideoExtentStait> ■ 

<VideoExtentEnd>25<yVideoExtentEnd> j 

i 

<RectImageExtent> | 

i 

<RectImageExtentX0>200<^ectIma^eExterLtX0> 
<^ectImageExtentYO>150</RectIma^eExt6iitY0> 
<RectImageExtentHeight>80</RectIi)iageExtentHeight> 
<^ectImageExtentWidtt^^ 
</RectImageExtent> j 
</VideoExtent> j 
</Locator> | 
</ AFLGamcDescription > i 
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Appendix D: Digital Video Resource Description Scheme 
Description Scheme (Video.ddf) 



<!- Core,ddf included here -> 

<! ENTITY % Core SYSTEM "Core,ddf > 

%Core; 



<!- 



Scheme specific entities 



<' ENTITY VxdeoDescGea SYSTEM "VideoDesc£en,class" NDATA 

i 

JAVACLASS> ! 

<! ENTITY ShotAnalyser SYSTEM "ShotAnalyser.class" NDjATA JAVACLASS> 

<! ENTITY VideoPresRules SYSTEM * e VideoPresentationRujies.xml"> 



<l- 



Video resource related element definitions 



J>1 



<! ELEMENT VideoDescription (Title, Shot*, Locator*)> | 
< ! ATTLIST VideoDescription ! 

superElement NMTOKEN #FIXED "Description" 

handler ENTITY #FIXED "VideoDescGen" 

ruleSets ^^^-^E^ #FIXED "VijieoPresRules" 

> D2 ' 
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<! ELEMENT Title (#PCDATA)> 








<!ATTLIST Title 








superElement NMTOKEN 

> 


#FD05D "Descriptor" 

i 




5 


<!ELEMENT Shot (Descriptor*)> 


i 






<!ATTLIST Shot ^ ■ 


" D3 ! 






superElement NMTOKEN 


#FDCED "Descriptor" 






handler . ENTITY 


#FDCED "ShotAnalyser" 






keyFrame\ ENTITY 


#REQUIRED ; 




10 


locator X. IDREF 


#REQUIRED; 






> D4 


i 
1 






<l ELEMENT PeopleScene (EMPTY)> 


! 






<!ATTLIST PeopleScene^^ D5 








superElement NMTOKEN 


#FIXED "Descriptor" 




15 


> 


i 






<!ELEMENT CrowdScene (EMPTY)> 








<! ATTLIST CrowdScene 


I 






superElement NMTOKEN 

> 


#FDCED "PecipleScene" 

r 




20 


<!ELEMENT PortraitScene (EMPTY)> 


i 






<! ATTLIST PortraitScene 


j 






superElement NMTOKEN 

> 


#FIXED "PeopieScene" 






<! ELEMENT IndoorScene (EMPTY) 


i 

i 




25 


<! ATTLIST IndoorScene 


i 

i 

! 






superElement NMTOKEN 

> 


#FIXED "Descriptor" 






<! ELEMENT OutdoorScene (EMPTY) 








<! ATTLIST OutdoorShot 






30 


superElement NMTOKEN 


#FIXED "Descriptor" 






> 
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Appendix E: An Example Description generated from the Video 
Description Scheme in Appendix D 

Example Description (VideoEg 1 .xml) 



<?xml version="l .0" standalone = u no" ?> , 
<!DOCTYPE VideoDescription SYSTEM "Video.ddf ' [ 

<! ENTITY MyVideo SYSTEM "MyVideo .mpg" NDATA MPEG2> 
<! ENTITY KFramel SYSTEM ''KFranu&jpg" NDATA JPEG> 
<!ENTITY KFrame2 SYSTEM "KFrame2.jpg" NDATA JPEG> 
etc. I 

]> j 
<VideoDescription resource ■* "MyVideo s > I 
<Title> Video Clip Title</Title> ' 
<r„ Shots detected in the digital video resource — > 
<Shot id = "SI" keyFrame = "KFramel" locator - *TJ "> 
<CrowdScene/> 

<OutdoorScene/> ! 
</Shot> ! 
<Shot id - "S2" keyFrame = "KFrame2" locator = "li"> 

<PortraitScene/> 

<OutdoorScene/> 
</Shot> ; 

<!— Locators in the digital video resource — > ! 
<Locator id = "LI" resource = "MyVideo"> ! 
<VideoExtent > 

<VideoExtentStart>0</VideoExtentStart> 
<VideoExtentEnd>20<yvideoExtentEnd> 
</VideoExtent> 
</Locator> 

<Locator id = U L2"> ; 
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<VideoExtent > 

<VideoExtentStart>2 1 </VideoExtentStart> 
<VideoExtentEnd>5CK/VideoExtentEnd> 
</VideoExtent> 
</Locator> 
</VideoDescription> 



CFPI595AU !PR34-3fi_GRPi 490568 



I:VELEC\CISRAMPRMPK34-38_ORP I U90S<*8 -doc P WM 



-120- 

i 
i 

Appendix F: Presentation Rules for the Video Description Scheme in 

i 

Appendix D 

Example Description (VideoPrescntationRules.xml) 



10 



15 



20 



25 



30 



<?xml version="l .0" standalone = "no" ?> 

<!DOCTYPE PresentationRules SYSTEM <c Rules.dtd" [ 

<! ENTITY CrowdScene SYSTEM "Cro wdScenelconjpg" NDATA JPEG> 
<!ENTITY PortraitScene SYSTEM fie PortraitSceneIconjpg >7 NDATA JPEG> 
<! ENTITY OutdoorScene SYSTEM "OutdoorScenelcoii jpg" NDATA JPEG> 
- <!ENTITY IndoorScene SYSTEM "IndoorScenelcon.jp^" NDATA JPEG> 

]> 

<Pre$entationR.ules> 

<Rule target = ElementDefh pattern = et VideoDescription* ! > 

i 

<Action> 

i 

<AddAttributeDef 

attName = "selected 7 * 
attType - "CDATA" j 
attDefault = W7> > 
</Action> ! 
<Action> ' 
<AddAttributeDef 

attName = "presentationType" j 
attType = "<lndex|TOCr j 
attDefault = #FDCED "TOC'7> 
</Action> | 
</Rulc> 1 
<Rule target = ElementDefh pattern = "VideoDescription/Shot"; 
<Action> 

<AddAttributeDef 

attName = "selected" ! 
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attType = "CDATA" 

attDefault = fiC NO"/> ; 

</Action> 
<Action> 

<AddAttributeDef 

attName — "presentationType 77 

attType -"(Index|TOCr 
attDefault = #FDCED "TOC"/> j 
</Action> | 
</Rule> | 

<Rule target - ElementDefh pattern ="VideoDescription/Shot/CrowdScene"> 
<Action> 

<AddAttributeDef 

attName — "presentationType" j 
attType = "(Index|TOC)' 7 | 
attDefault - #FDOED "Index'7> 

</Action> 
<Action> 

<AddAttributeDef 

attName = "icon" j 

attType - "ENTITY" j 

attDefault = #FEXED <6 CrowdScene'Y> 

</Action> 
</Rule> 



25 
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<Rule targets ElementDefh pattern ="VideoDescription/Shot/'PortraitScene"> 
<Action> 

<AddAttributeDef 

attName = '"presentationType" 
attType - "(IndexITOC)" 
attDefault - #PIXED "Index"/> 

f 

</Action> 
<Action> 

<AddAttributeDef 
10 attName « "icon" 

attType = ENTITY" 

attDefault = #FD0ED "PortraitScencf /> 

i 

<yAction> ; 
</Rule> ! 
15 <Rule target = ElementDefh pattern = 

'VideoDescription/Shot/IndoorScene^ 
<Action> 

<AddAttributeDef 

i 

attName = "presentationType" j 
20 attType = "(Index|TOC)" j 

attDefault - #FDtED "Index* V> 

</Action> 
<Action> 

<AddAttributeDef [ 
25 attName = "icon" ! 

attType = "ENTITY" j 
attDefault = #FDCED "IndoorScene 5i 7> 

<VAction> 
</Rule> 



i 
i 
i 
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<Rule target=ElementDefii pattern 
"VideoDescriptioii/Shot/OiitdoorScene*^ 
<Action> 

<AddAttributeDef 

attName = "presentationType" 
attType = "(Tfodex|TOC)" 
attDefault - #FIXED "Index'7> 

</Action> 

<Action> 
<AddAttributcDcf 

attName = "icon" 
attType - "ENTITY" 



attDefault = #FIXED "Outdo orScene"/> 



<^Action> 
</Rule> 
</Presentati or)Rules> 



i 
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Appendix G: Digital Video Library Description Scheme 



Description Scheme (DigitalVideoLibrary.ddf) 



5 


<!— Core.ddf included here — > 








<! ENTITY % Core SYSTEM "Core.ddf > 








%Core; 


j 




10 


Scheme specific entities 








<! ENTITY VideoLibraryGen SYSTEM "VideoLibraryGen.class" 


> 




NDATA JAVACLASS> 

<-| _ _ __ 






-4 t— 

15 


Digital Video Library related element definitions ! 






<! ELEMENT DigitalVideoLibraryDescription 


(Section* | Item*)> 






<! ATTLIST DigitalVideoLibraryDescription 








supcrElement NMTOKEN 


#FIXED "Description" 




20 


handler ENTITY 


#FIXED "VideoLibraryGen" 






title CDATA 

> 


#IMPLIED ! 






<! ELEMENT Section (Section* | Item*)> 








<!ATTLIST Section 






25 


superElem ent NMTOKEN 


#FIXED "Descriptor" 






title CDATA 


#IMPLIED 






> 







C1-P1595AU IPR34-3B_GRP1490568 



r:\ELEC\CISRA\IPRVlPR34-38_GRP 1N49056K docPWM 



-125- 



<! ELEMENT Item (EMPTY)> 








superElement NMTOKEN 


#FIXED "Descriptor" 


description ENTITY 


#REQU1RED 


> 





i 



I 
I 

I 

I 

[ 



I 

i 
i 

! 

! 
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Appendix H: An Example Description generated frofri the Digital Video 
Library Description Scheme in Appendix G 

Example Description (VideoLibraryEg.xml) 

5 

i 

<?xml version^" 1.0" standalone = "no" ?> 

<!DOCTYPE DigitalVideoLibraryDescription SYSTEM "DigitalVideoLibrary.ddf' [ 
<!ENTITY VideoEgl SYSTEM "VideoEgl jcml'> 
<! ENTITY VideoEg2 SYSTEM "VideoEg2.xml"> 

<! ENTITY VideoEg3 SYSTEM "VideoEg3 .xml"> ! 

i 

etc. 

]> : 

<DigitalVideoLibraryDescription title = "My Personal Digital Video Library"> 
<Section title = "Holiday Videos"> ; 
<Item description = " VideoEg 1' 7> ■ 
<Item description = t< VideoEg2"/> 
etc. | 
</Section> 

i 

<Section title "Birthday Videos"> 

<Section title = A *Mary 7 s Birthdays"> 

<Item description = "VideoEg3/> \ 

i 

etc. 

</Seciion> ! 
<Section title = "John's Birthdays"> ... </Secti6n> 
</Sectjon> 
</DigitalVideoLibraryDescription> 
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Appendix I; Video Presentation Description Scheme 
Description Scheme (VideoPresentation.d^f) 



<!-- Core. ddf included here --> 

<! ENTITY % Core SYSTEM "Core.ddf"> 

%Core; 

<» _„ 



Scheme specific entities 



<! ENTITY VideoPresentationGen SYSTEM "VideoPresenUtion.class" 

ND ATA JAVACLASS> 
<! ENTITY VideoPresentationRules SYSTEM "VideoPreseniionRules.xmr> 

<! . : 

Video Presentation related element definitions 



<!ELEMENT VideoPresentationDescription (VideoDescriptibnReference*)> 
<!ATTLIST VideoPresentationDescription 

superElement NMTOKEN #FIXED "Description" 

handler ENTITY 
title CDATA 
ruleS ets ENTITIES 
userPresentationRules ENTITY 

<! ELEMENT VideoDescriptionReference (ShotRefcrence*) , 
<!ATTLIST VideoDescriptionReference , 

superElement NMTOKEN #FIXED "Descriptor" 

videoDescription ENTITY #REQUIRED 



#FDOED "VideoPresentationGen" 
#IMPLIED : 

#FIXED "VikeoPresentationiRules" 
#1MPLIED ; 
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<! ELEMENT ShotReference (EMPTY) 


i 
i 


<!ATTLIST ShotReference 








shotlDRef IDREF 


#REQUIRE]j> 

i 


> 


1 



I 



i 



! 



I 



; 



i 

! 



I 
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Appendix J; An Example Description generated from the Video 
Presentation Description Scheme in Appendix I 

Example Description (VideoPresentationEg.xml) 



<?xml version="l .0" standalone - "no" ?> 

<!DOCTYPE VideoPresentationDescription SYSTEM "VideoPresentation.ddf ' [ 
<!ENTITY UserPresentationRules SYSTEM "UserPrisentationRules.xmr'> 
<! ENTITY VideoEgl SYSTEM "VideoEgl .xml"> i 
<!ENTITY VideoEg2 SYSTEM "VideoEg2.xml"> 
etc. 

]> 

i 

<VideoPresentationDescription userPresentationRules = *'Us6rPresentationRules"> 
<VideoDescriptionReference videoDescription - "VideoEgl"> ^ 

<ShotRcfcrence shotlDRef - "S l"/> ^ „J2 Jl 

<ShotReference shotlDRef = "S2"/> 
</ VideoDescriptionReference > 



<VideoDescriptionRefexenoe description = "VideoEg^"> 
<ShotReference shotlDRef = "S3"/> ! 
<ShotReference shotlDRef = "S4"/> j 

</ VideoDescriptionReference > 



etc. 



</VideoPresentationDescription> 
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Appendix K: DOM Element Nodes 

Extract from DOM Version 1.0 obtained on the Website 
http://www.w3 .org/TR/1998/REC-DOM-level- 1 - i 99 8 1 000 1 

Interface Element 

By far the vast majority (apart from text) of node types that authors will generally 
encounter when traversing a document will be Element nodes. These objects represent 
both the element itself, as well as any contained notes. For example (in XML): 
<elementExample id="demo"> 
<sub element 1 /> 

<subelement2><^ubsubelement/></subelement2> 
</elementExample> 



When represented using DOM, the top node would be "ejementExample", which 
contains two child Element nodes (and some space), one for "subelementl" and one for 
"subelement2". "subelementl" contains no child nodes of its own. 



interface Element: 


Node { 


wstring 


getTagNameQ; 


Nodelterator 


get AttributesO; 


wstring 


get Attribute (in name name); 


void 


setAttribute (in string name, in string value); 


void 


removeAttribute(in wstring name); 


Attribute 


getAttributeNode(m name name); 1 


void 


setAttributeNode(in Attribute newAttr); 


void 


removeAttributeNode(in Attribute oldAttr); 


void 


getElementsByTagName(in wstring tagname); 


void 

h 


normalize(); 



Method getTagNameQ 

This method returns the string that is the element's name. For example, in: 
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<elementExample id-'demo 1 ^ 
</elementExample> 



This would have the value "elementExample", Note that this is case-preserving, as 
are all of the operations of the DOM, See Name case in the DOM for a description why 
the DOM preserves case. 

Parameters 

This method has no parameters ; 



Return Values 



wstring 



10 Exceptions 



This method throws no exceptions ! 



15 



Method getAttributesO 

The attributes for this element. In the elementExample example above, the 
attributes list would consist of the id attribute, as well as any attributes which were 
defined by the document type definition for this element which have default values. 



Parameters 



This method has no parameters 



20 



Return Values 



Nodelterator 



Exceptions 



This method throws no exceptions 



Method getAitributeQ 
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Retrieves an Attribute value by name from an Element, object. 
Parameters 



name 



The name of the attribute to retrieve 



Return Values 



wstring 



Exceptions 



This method throws no exceptions 



Method setAttributeQ 

Adds a new attribute/value pair to an Element node object If an attribute by that 
name is already present in the element, its value is changed to thai of the value parameter. 

i 

Parameters 



name 




value 





Return Values 



void 



Exceptions 



This method throws no exceptions 



Meth od rem oveA UributeQ 

Removes the specified attribute from an Element node object. 

Parameters 



name 



Return Values 
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void 



Exceptions 



This method throws no exceptions 



Method getAMributeNodeQ 
5 Retrieves an Attribute node by name from an Element, object. 



Parameters 



name 



The name of the attribute to retrieve 



Return Values 






Attribute 






10 







Exceptions 



This method throws no exceptions 



Method setAttributeNode() 

Adds a new attribute/value pair to art Element node object If an attribute by that 
15 name is already present in the element, its value is changed toibe that of the Attribute 
instance. 



Parameters 



newAttr 



20 Return Values 



void 



Exceptions 



This method throws no exceptions 



Method removeAttributeNodeQ 
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Rcmoves the specified attribute/value pair from an Element node object. 
Parameters 

"oldAttr ™ j 

1 - . ! 

Return Values 

void ~~ " ; ^ 



Exceptions 

This method throws no exceptions 



Meth od getElement&ByTagNameO 

Returns an iterator through all subordinate elements with a given tag name. 

Parameters 

tagname I 



Return Values 
void 



Exceptions 

This method throws no exceptions 



Method normalizeQ 

Puts all Tet nodes in the sub-tree underneath this Element into a "normal" form 
where only markup (eg tags, comment, Pis, CDATASections) and entity references 
separate Text nodes. ; 



Parameters 



This method has no parameters 



Return Values 
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void 



Exceptions 



This method throws no exceptions 



Attribute data 

This holds the actual content of the text node. Text nodes contain just plain text, 
without markup and without entities, both of which are represented as separate objects in 
the DOM. 
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CLAIMS: 

1 . A method of browsing electronically-accessible resources using descriptions of said 
resources, wherein said descriptions of said resources have descriptor components, said 

5 descriptor components having attributes representative of at least iwo axes of access to the 
resources and wherein said descriptions have links to corresponding said electronically- 
accessible resources, said method comprising the steps of: 
reading said descriptions; 

displaying items, wherein each item is associated witty a corresponding said 
10 descriptor component of a said read description that has at least oijie said attribute; and 

browsing said descriptions of the resources and their correjsponding electronically- 
accessible resources via said links using said displayed items, ! 

2. A method as claimed in claim 1, wherein each said read description is represented 
15 by a tree of descriptor components, and one or more of said descriptor components have 

descriptor components as descendents. j 

| 

3. A method as claimed in claim 1, wherein one of said axes of access is a table-of- 
contents classification. 

20 

4. A method as claimed in claim 1, wherein one of said axes of access is an index 
classification. j 

5. A method as claimed in claim 1, wherein the descriptions of the resources have 
25 been generated using a description scheme as a template, and the description scheme uses 

a declarative description definition language which contains definitions for descriptor 
components of the said descriptions of the resources. 

j 

6. A method as claimed in claim 5, wherein said attributes of the descriptor 
30 components are defined in the description scheme. 

7. A method as claimed in claim 5, wherein the said attributes of the descriptor 
components are a persistent item of the description scheme. 
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8. A method as claimed in claim 5 y wherein the said attributes of the descriptor 
components are instantiated by an application when required. 

I 

9. A method as claimed in claim 8, wherein the said attributes of the descriptor 
components are instantiated using a rule that is associated with the description scheme. 

10. A method as claimed in claim 1, wherein the resource is an item of digital content. 

11. A method as claimed in claim 1, wherein the resource is an electronic document or 
resource available over the World Wide Web, 

12. A method as claimed in claim 1, wherein the resource is an Electronic device. 

13. A method as claimed in claim 1, wherein the description of the resource contains 
links to identified sections of the resource. 

14. A method as claimed in claim 1 ? wherein said axes of adcess are determined by 
rules operating on the description. 

15. A method as claimed in claim 1, wherein said axes of accesjs are determined during 
the generation of the description of the resource. ! 

16. A method as claimed in claim 1, wherein said attributes of said descriptor 
components representative of said at least two axes of access are inferred from the content 
of the description. \ 

i 

17. A method as claimed in claim 16, wherein said attribute of a said descriptor 
component is inferred to be a table of content descriptor if the said descriptor component 
contains a reference to a resource or a section of a resource. 
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18. A method as claimed in claim 17, wherein said attribute of a said descriptor 
component is inferred to be an index descriptor if !he said descriptor component is not 
inferred to be a table of contents descriptor. 

19. A method of searching electronically-accessible resources using descriptions of said 
resources, wherein said descriptions of said resources have descriptor components, said 
descriptor components having attributes representative of at least two axes of access to the 
resources and wherein said descriptions have links to corresponding said electronically- 
accessible resources, said method comprising the steps of: 

reading said descriptions; 

displaying items, wherein each item is associated with a corresponding said 
descriptor component of a said read description that has at least one said attribute; 

specifying a query in terms of index descriptors; 

searching said descriptions of the resources using said query; and 

locating said corresponding electronically-accessible resources using said links 
associated with said displayed items. 

20. A method as claimed in claim 19, wherein each said read description is represented 
by a tree of descriptor components, and one or more of said descriptor components have 
descriptor components as descendents. 

21. A method as claimed in claim 19, wherein said method further comprises the step 
of 

playing resources corresponding to said located descriptions of resources. 

22. A method as claimed in claim 19, wherein said attributes of said descriptor 
components representative of said at least two axes of access are inferred from the content 
of the description. 

23. A method as claimed in claim 22, wherein said attribute of a said descriptor 
component is inferred to be a table of content descriptor if the said descriptor component 
contains a reference to a resource or a section of a resource. 
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24, A method as claimed in claim 23, wherein said attribute of a said descriptor 
component is inferred to be an index descriptor if the said descriptor component is not 
inferred to be a table of contents descriptor. 

5 25, A method of locating electronically-accessible resources using descriptions of said 
resources, wherein the descriptions of said resources have descriptor components, each 
said descriptor component comprises the association of a feature of a said resource with a 
representative value for that feature, and one or more of said ; descriptor components 
including a table of contents attribute and one or more of said descriptor components 
10 including an index attribute, and wherein said descriptions have links to corresponding 
said electronically- accessible resources, said method comprising the steps of: 
reading said descriptions; ; 

displaying a table of contents containing table of contents items, wherein each table 
of contents item is associated with a corresponding said descriptor component that has a 
15 table of contents attribute; 

selecting one said displayed table of contents item; 

displaying an index containing index items, wherein each said displayed index item 
is associated with a corresponding said descriptor component that has an index attribute 
and that is associated with the said selected table of contents item; ' 
20 selecting one or more index items in the displayed index; 

specifying a said representative value or values for the one or more said, selected 
index items; 

searching said descriptions of the resources for said one or more selected index 
items and their corresponding said specified representative value or values; and 
25 locating one or more said descriptions of the resources corresponding to said one ox 

more selected index items and their corresponding said specified Representative value or 
values. 

26. A method as claimed in claim 25, wherein each said read description is represented 
30 by a tree of descriptor components, and one or more of said descriptor components have 
descriptor components as descendents. 
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27. A method as claimed in claim 25, wherein said method further comprises the step 
of: 

playing resources corresponding to said located descriptions of resources. 

5 28, A method as claimed in claim 25, wherein said step of selecting one said table of 
contents item is optional and if not performed said step of displaying an index displays all 
said index items associated with all said table of contents items. 

29. A method as claimed in claim 25, wherein said selecting step of index items 
10 comprises selecting logical combinations of said displayed index items and their said 

representative values. 

30. A method as claimed in claim 25, wherein said attribute of a said descriptor 
component is inferred to be a table of content descriptor if the said descriptor component 

15 contains a reference to a resource or a section of a resource. 

31. A method as claimed in claim 30, wherein said attribute of a said descriptor 
component is inferred to be an index descriptor if the said descriptor component is not 
inferred to be a table of contents descriptor. 

20 

32. A method of annotating an electronically-accessible resource using a description of 
said resource, wherein the description of said resource has descriptor components, each 
said descriptor component comprises the association of a feature of said resource with a 
representative value for that feature, and one or more of said descriptor components 

25 including a table of contents attribute and one or more of said descriptor components 
including an index attribute, said method comprising the steps of: 
reading said descriptions; 

displaying a table of contents containing table of contents items, wherein each table 
of contents item is associated with a corresponding said descriptor component that has a 
30 table of contents attribute; 

selecting one said displayed table of contents item for the annotation; 
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displaying an index containing index items, wherein each said displayed index item 
is associated with a corresponding said descriptor component that has an index attribute 
and that is associated with the said selected table of contents item; 

selecting one said displayed index item; 
5 associating said selected displayed index item with said selected table of contents 

item; 

choosing a said representative value for the selected index item; and 
associating said chosen representative value with said feature which corresponds to 
said selected index item, wherein said chosen representative value and its corresponding 
10 feature provide an annotation of the resource. 

33. A method as claimed in claim 32, wherein each said read description is represented 
by a tree of descriptor components, and one or more of said descriptor components have 
descriptor components as descendents. 

15 

34, A method as claimed in claim 32, wherein said step of associating said selected 
display index item is allowed only if the corresponding descriptor of said selected display 
index item is a valid descriptor for the table of contents item selected for annotation. 

20 35. A method as claimed in claim 32, wherein said step of choosing a said 
representative value is predetermined. 

36. Apparatus for browsing electronically-accessible resources using descriptions of 
said resources, wherein said descriptions of said resources have descriptor components, 
25 said descriptor components having attributes representative of at least two axes of access 
to the resources and wherein said descriptions have links to corresponding said 
electronically- accessible resources, said apparatus comprising; 
means for reading said descriptions; 

means for displaying items, wherein each item is associated with a corresponding 
30 said descriptor component of a said read description that has at least one said attribute; 
and 

means for browsing said descriptions of the resources and their corresponding 
electronically-accessible resources via said links using said displayed items. 
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37. Apparatus as claimed as claimed in claim 36, wherein said means for reading said 
descriptions represents each description by a tree of descriptor components, and one or 
more of said descriptor components have descriptor components as descendants. 

5 

38. Apparatus as claimed in claim 36, wherein one of said axes of access is a table-of- 
contents classification. 

39. Apparatus as claimed in claim 36, wherein one of said axes of access is an index 
10 classification. 

40. Apparatus as claimed in claim 36, wherein the descriptions of the resources have 
been provided using a description scheme as a template, and the description scheme uses 
a declarative description definition language which contains definitions for descriptor 

15 components of the said descriptions of the resources. 

41. Apparatus as claimed in claim 40, wherein said attributes of the descriptor 
components are defined in the description scheme. 

20 42. Apparatus as claimed in claim 40, wherein the said attributes of the descriptor 
components are a persistent item of the description scheme, 

43. Apparatus as claimed in claim 40 3 wherein the said attributes of the descriptor 
components are instantiated by an application when required. 

25 

44. Apparatus as claimed in claim 43, wherein the said attributes of the descriptor 
components are instantiated using a rule that is associated with the description scheme. 

45. Apparatus as claimed in claim 36, wherein the resource is an item of digital 
30 content. 

46. Apparatus as claimed in claim 36, wherein the resource is an electronic document or 
resource available over the World Wide Web. 
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47. Apparatus as claimed in claim 36, wherein the resource is an electronic device. 

48. Apparatus as claimed in claim 36, wherein the description of the resource contains 
5 links to identified sections of the resource. 

49. Apparatus as claimed in claim 36, wherein said axes of access are determined by 
rules operating on the description. 

10 50. Apparatus as claimed in claim 36, wherein said axes of access are determined 
during the generation of the description of the resource. 

51. Apparatus as claimed in claim 36, wherein said attributes of said descriptor 
components representative of said at least two axes of access are inferred from the content 

15 of the description. 

52. Apparatus as claimed in claim 51, wherein said attribute of a said descriptor 
component is inferred to be a table of content descriptor if the said descriptor component 
contains a reference to a resources or a section of a resource. 

20 

53. Apparatus as claimed in claim 52, wherein said attribute of a said descriptor 
component is inferred to be an index descriptor if the said descriptor component is not 
inferred to be a table of contents descriptor. 

25 54, Apparatus for searching electronically-accessible resources using descriptions of 
said resources, wherein said descriptions of said resources have descriptor components, 
said descriptor components having attributes representative of at least two axes of access 
to the resources and wherein said descriptions have links to corresponding said 
electronically- accessible resources, said apparatus comprising: 
30 means for reading said descriptions; 

means for displaying items, wherein each item is associated with a corresponding 
said descriptor component of a said read description that has at least one said attribute; 
means for specifying a query in terms of index descriptors; 
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means fox searching said descriptions of the resources using said query; and 
means for locating said corresponding electronically-accessible resources using said 
links associated with said displayed items. 

5 55. Apparatus as claimed in claim 54, wherein said read means represents each said 
description by a tree of descriptor components, and one or more of said descriptor 
components have descriptor components as descendents. 

56. Apparatus as claimed in claim 54, wherein said apparatus further comprises: 
10 means for playing resources corresponding to said located descriptions of resources. 

57. Apparatus as claimed in claim 54, wherein said attributes of said descriptor 
components representative of said at least two axes of access are inferred from the content 
of the description, 

15 

58. Apparatus as claimed in claim 57> wherein said attribute of a said descriptor 
component is inferred to be a table of content descriptor if the said descriptor component 
contains a reference to a resources or a section of a resource. 

20 59. Apparatus as claimed in claim 58, wherein said attribute of a said descriptor 
component is inferred to be an index descriptor if the said descriptor component is not 
inferred to be a table of contents descriptor. 

60. Apparatus for locating electronically-accessible resources using descriptions of said 
25 resources, wherein the descriptions of said resources have descriptor components, each 
said descriptor component comprises the association of a feature of a said resource with a 
representative value for that feature, and one or more of said descriptor components 
including a table of contents attribute and one or more of said descriptor components 
including an index attribute, and wherein said descriptions have links to corresponding 
30 said electronically- accessible resources, said apparatus comprising: 
means for reading said descriptions; 
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means for displaying a table of contents containing table of contents items, wherein 
each table of contents item is associated with a corresponding said descriptor component 
that has a table of contents attribute; 

means for selecting one said displayed table of contents item; 
5 means for displaying an index containing index items, wherein each said displayed 

index item is associated with a corresponding said descriptor component that has an index 
attribute and that is associated with the said selected table of contents item; 

means for selecting one or more index items in the displayed index; 

means for specifying a said representative value or values for the one or more said 
10 selected index items; 

means for searching said descriptions of the resources for said one or more selected 
index items and their corresponding said specified representative value or values; and 

means for locating one or more said descriptions of the resources corresponding to 
said one or more selected index items and their corresponding said specified 
15 representative value or values. 

61. A method as claimed in claim 60, wherein said read means represents each said 
description by a tree of descriptor components, and one or more of said descriptor 
components have descriptor components as descendents. 

20 

62. Apparatus as claimed in claim 60, wherein said apparatus further comprises:, 
means for playing resources corresponding to said located descriptions of resources. 

63. Apparatus as claimed in claim 60, wherein operation of said means for selecting one 
25 said table of contents item is optional and if not performed said means for displaying an 

index displays all said index items associated with all said table of contents items. 

64. Apparatus as claimed in claim 60, wherein said means for selecting said 
representative value or values comprise means for selecting logical combinations of said 

30 displayed index items and their said representative values. 
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65. Apparatus as claimed in claim 64, wherein said attribute of a said descriptor 
component is inferred to be a table of content descriptor if the said descriptor component 
contains a reference to a resources or a section of a resource. 

5 66. Apparatus as claimed in claim 65, wherein said attribute of a said descriptor 
component is inferred to be an index descriptor if the said descriptor component is not 
inferred to be a table of contents descriptor. 

67. Apparatus for annotating an electronically-accessible resource using a description of 
10 said resource, wherein the description of said resource has descriptor components, each 

said descriptor component comprises the association of a feature of said resource with a 
representative value for that feature, and one or more of said descriptor components 
including a table of contents attribute and one or more of said descriptor components 
including an index attribute, said apparatus comprising: 
1 5 means for reading said descriptions; 

means for displaying a table of contents containing table of contents items, wherein 
each table of contents item is associated with a corresponding said descriptor component 
that has a table of contents attribute; 

means for selecting one said displayed table of contents item for the annotation; 
20 means for displaying an index containing index items, wherein each said displayed 

index item is associated with a corresponding said descriptor component that has. an index 
attribute and that is associated with the said selected table of contents item; 

means for selecting one said displayed index item; 

means for associating said selected displayed index item with said selected tabic of 
25 contents item; 

means for choosing a said representative value for the selected index item; and 
means for associating said chosen representative value with said feature which 
corresponds to said selected index item, wherein said chosen representative value and its 
corresponding feature provide an annotation of the resource. 

30 

68. Apparatus as claimed in claim 67, wherein said read means represents each said 
description by a tree of descriptor components, and one or more of said descriptor 
components have descriptor components as descendents. 
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69, Apparatus as claimed in claim 67, wherein operation of said means for associating 
said selected display index item is allowed only if the corresponding descriptor of said 
selected display index item is a valid descriptor for the table of contents item selected for 
5 annotation. 



70. Apparatus as claimed in claim 67 7 wherein operation of said means for selecting one 
said table of contents item is optional aad if not performed said means for displaying an 
index displays all said index items associated with all said table of contents items. 

10 

71, A computer readable medium comprising a computer program for browsing 
eJectronically-accessible resources using descriptions of said resources, wherein said 
descriptions of said resources have descriptor components, said descriptor components 
having attributes representative of at least two axes of access to the resources and wherein 

15 said descriptions have links to corresponding said electronically- accessible resources, 
said computer program comprising: 

code for reading said descriptions; 

code for displaying items, wherein each item is associated with a corresponding said 
descriptor component of a said read description that has at least one said attribute; and 
20 code for browsing said descriptions of the resources and their corresponding 

electronically-accessible resources via said links using said displayed items. 



72. A computer readable medium comprising a computer program for searching 
electronically-accessible resources using descriptions of said resources, wherein said 
25 descriptions of said resources have descriptor components, said descriptor components 
having attributes representative of at least two axes of access to the resources and wherein 
said descriptions have links to corresponding said electronically- accessible resources, 
said computer program comprising: 

code for reading said descriptions; 
30 code for displaying items, wherein each item is associated with a corresponding said 

descriptor component of a said read description that has at least one said attribute; 

code for specifying a query in terms of index descriptors; 

code for searching said descriptions of the resources using said query; and 
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code for locating said corresponding electronically-accessible resources using said 
links associated with said displayed items, 

73. A computer readable medium comprising a computer program for locating 
5 electronically-accessible resources using descriptions of said resources, wherein the 

descriptions of said resources have descriptor components, each said descriptor 
component comprises the association of a feature of a said resource with a representative 
value for that feature, and one or more of said descriptor components including a table of 
contents attribute and one or more of said descriptor components including an index 
10 attribute, and wherein said descriptions have links to corresponding said electronically- 
accessible resources, said computer program comprising: 
code for reading said descriptions; 

code for displaying a table of contents containing table of contents items, wherein 
each table of contents item is associated with a corresponding said descriptor component 
15 that has a table of contents attribute; 

code for selecting one said displayed table of contents item; 

code for displaying an index containing index items, wherein each said displayed 
index item is associated with a corresponding said descriptor component that has an index 
attribute and that is associated with the said selected table of contents item; 
20 code for selecting one or more index items in the displayed index; 

code for specifying a said representative value or values for the one or more said 
selected index items; 

code for searching said descriptions of the resources for said one or more selected 
index items and their corresponding said specified representative value or values; and 
25 code for locating one or more said descriptions of the resources corresponding to 

said one or more selected index items and their corresponding said specified 
representative value or values. 

74. A computer readable medium comprising a computer program for annotating an 
30 electronically-accessible resource using a description of said resource, wherein the 

description of said resource has descriptor components, each said descriptor component 
comprises the association of a feature of said resource with a representative value for that 
feature, and one or more of said descriptor components including a table of contents 
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attribute and one or more of said descriptor components including an index attribute, said 
computer program comprising: 

code for reading said descriptions; 

code for displaying a table of contents containing table of contents items, wherein 
5 each table of contents item is associated with a corresponding said descriptor component 
that has a table of contents attribute; 

code for selecting one said displayed table of contents item for the annotation; 
code for displaying an index containing index items, wherein each said displayed 
index item is associated with a corresponding said descriptor component that has an index 
10 attribute and that is associated with the said selected table of contents item; 
code for selecting one said displayed index item; 

code for associating said selected displayed index item with said selected table of 
contents item; 

code for choosing a said representative value for the selected index item; and 
15 code for associating said chosen representative value with said feature which 

corresponds to said selected index item, wherein said chosen representative value and its 
corresponding feature provide an annotation of the resource. 

75. A method of selecting one or more descriptions or one or more descriptor 
20 components from a set of descriptions, wherein said descriptions comprise one or more 
said descriptor components, and each said description of said set of descriptions is 
associated with a corresponding electronically accessible resource, said method 
comprising the steps of: 

specifying a desired selection of descriptor components; 
25 generating a selection rule based on said specified descriptor components, wherein 

said selection rule having a predetermined pattern and action component, wherein the 
said predetermined pattern represents a specified pattern of descriptor components and the 
said action specifies the action to be performed when a said descriptor component in the 
descriptions of the set of descriptions matches the predetermined pattern; 
30 reading said descriptions of the resources; 

locating patterns of descriptor components in descriptions of said set of 
descriptions in order to select descriptor components or descriptions in said set of 
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descriptions having said descriptor components which match the predetermined pattern; 
and 

performing said specified action. 

5 76. The method as claimed in claim 75 5 wherein each said read description is represented 
by a tree of descriptor components, and one or more said descriptor components have 
descriptor components as descendents. 

77. The method as claimed in claim 76, wherein said predetermined pattern is 
10 represented using the context of said tree of descriptor components. 

78. The method as claimed in claim 77, wherein the said action component of the 
selection rule involves setting the value of a predetermined descriptor attribute in order to 
indicate which descriptor components have been selected to an application. 

15 

79. The method as claimed in claim 76, wherein the said predetermined descriptor 
attribute is also set for each of the ancestors of a descriptor component which matches the 
said predetermined pattern. 

20 80. The method as claimed in claim 76, wherein the said predetermined descriptor 
attribute is also set for the root descriptor of the tree containing a descriptor which 
matches the said predetermined pattern. 

81. The method as claimed in claim 80, wherein the setting of the predetermined 
25 descriptor attribute for the root descriptor of the tree corresponds to the selection of the 

description. 

82. The method as claimed in claim 79, wherein the said pattern component of the said 
selection rule is automatically set to the current structural context set by a browsing 

30 application. 

83. Apparatus for selecting one or more descriptions or one or more descriptor 
components from a set of descriptions, wherein said descriptions comprise one or more 
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said descriptor components, and each said description of said set of descriptions is 
associated with a corresponding electronically accessible resource, said apparatus 
comprising: 

means for specifying a desired selection of descriptor components; 
5 means for generating a selection rule based on said specified descriptor 

components, wherein said selection rule having a predetermined pattern and action 
component, wherein the said predetermined pattern represents a specified pattern of 
descriptor components and the said action specifies the action to be performed when a 
said descriptor component in the descriptions of the set of descriptions matches the 
10 predetermined pattern; 

means for reading said descriptions of the resources; 

means for locating patterns of descriptor components in descriptions of said set of 
descriptions in order to select descriptor components or descriptions in said set of 
descriptions having said descriptor components which match the predetermined pattern; 
15 and 

means for performing said specified action. 

84 T A computer readable medium comprising a computer program for selecting one or 
more descriptions or one or more descriptor components from a set of descriptions, 

20 wherein said descriptions comprise one or more said descriptor components, and each 
said description of said set of descriptions is associated with a corresponding 
electronically accessible resource, said computer program comprising: 

code for specifying a desired selection of descriptor components; 

code for generating a selection rule based on said specified descriptor components, 

25 wherein said selection rule having a predetermined pattern and action component, 
wherein the said predetermined pattern represents a specified pattern of descriptor 
components and the said action specifies the action to be performed when a said 
descriptor component in the descriptions of the set of descriptions matches the 
predetermined pattern; 

30 code for reading said descriptions of the resources; 

code for locating patterns of descriptor components in descriptions of said set of 
descriptions in order to select descriptor components or descriptions in said set of 
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descriptions having said descriptor components which match the predetermined pattern; 
and 

code for performing said specified action. 

5 85. A method of generating on an output device a presentation based on a 
predetermined selection of resources, said method comprising the steps of: 

reading a description scheme for said presentation, wherein the description 
scheme for said presentation uses a declarative description definition language which 
contains definitions for descriptor components of the description scheme; 
10 generating a description of the said presentation using said description scheme 

and said predetermined selection of said resources; and 

generating on said output device the said presentation based on the said 
description of the said presentation and the predetermined selected resources. 



15 86. The method as claimed in claim 85, wherein said resources are digital video 
resources and said presentation is a video presentation, 

87. The method as claimed in claim 86, wherein said output device is a display device. 

20 88. The method as claimed in claim 85, wherein said description scheme has an 
associated reference to procedural code for the instantiation of a descriptor in the 
description of the presentation. 

89. The method as claimed in claim 86 ? wherein the said description of the video 
25 presentation can specify sections of different individual digital video resources to be used 

to render the video presentation. 

90. The method as claimed in claim 85, wherein the method comprises associating with 
the said description scheme a set of presentation rules. 

30 

91. The method as claimed in claim 90, wherein the presentation rules specify the type of 
transitions that are to be rendered between sections of the presentations. 
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92. The method as claimed in claim 90, wherein the presentation rules specify whether a 
title is rendered for the presentation. 

93. The method as claimed in claim 90, wherein the presentation rules specify the spatial 
5 placement, colour, font and/or size of a title to be rendered for the presentation. 

94. The method as claimed in claim 86, wherein the method comprises associating with 
the said description scheme a set of presentation rules and the presentation rules specify 
the speed at which components are played in the video presentation. 

10 

95. The method as claimed in claim 85, wherein the method further comprises 
associating with the said description scheme a set of presentation rules which specify 
characteristics of the style of the presentation for said description generated using the said 
description scheme. 

15 

96. The method as claimed in claim 95, associating with the said description a further set 
of presentation rules which specify characteristics of the style of the presentation to be 
generated from the said description. 

20 97. The method as claimed in claim 94, wherein the presentation rules specify that a 
particular component of a presentation is to be played at the original recorded. speed, a 
slower speed or a faster speed. 

98. The method as claimed in claim 85 ? wherein said resources are digital image 
25 resources. 

99. The method as claimed in claim 85, wherein said presentation is a printed 
presentation. 

30 100. The method as claimed in claim 85, wherein said output device is a printer device. 
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101. The method as claimed in claim 99, wherein the method comprises associating with 
the said description scheme a set of presentation rules and said presentation rules specify 
the spatial layout of the printed presentation. 

5 102. The method as claimed in claim 99, wherein the method comprises associating with 
the said description scheme a set of presentation rules and said presentation rules specify 
colour information for the printed presentation. 

103. The method as claimed in claim 98, wherein the said digital image resources 
10 contains image frames from digital video resources stored on a DVD. 

104. The method as claimed in claim 85, wherein the step of reading a description 
scheme and the step of generating a description are performed on a source device and said 
step of generating said presentation is performed on said output device and said method 

15 further comprises the step of: 

communicating the said description from the source to the output device via a 
wireless connection. 

105. The method as claimed in claim 104, wherein the resources are digital video 
20 resources which are stored and accessed from a processing server, which is separate from 

and has a wireless connection to the said source device, and a connection to, the said 
output device. 

106. The method as claimed in claim 105, wherein the said processing server renders the 
25 presentation for viewing from the description of the presentation and communicates the 

rendered presentation to the said output device. 

107. The method as claimed in claim 105, wherein the said processing server 
communicates both the description of the presentation and the associated digital video 

30 resources to the said output device which subsequently renders the presentation for 
viewing. 
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108. The method as claimed in claim 104, wherein the resources are digital video 
resources which are stored and accessed from the said source device and communicated 
with the said description of the presentation to the said output device, which has digital 
video display capability. 

5 

109. The method as claimed in claim 108, wherein a digital video resource associated 
with the description is communicated via the wireless connection in encoded format. 

110. The method as claimed in claim 108, wherein a digital video resource associated 
10 with the description is communicated via the wireless connection in compressed format, 

111. A method of generating on an output device a presentation based on a 
predetermined selection of resources, said method comprising the steps of: 

reading a description scheme for said presentation, wherein said description 
15 scheme contains definitions for descriptor components of the description scheme, and 

each said descriptor component comprises the association of a presentation attribute 

with a representative value for that attribute; 

generating a description of the said presentation using said description scheme 

and said predetermined selection of resources; and 
20 generating on said output device the said presentation based on the said 

generated description of the said presentation, the predetermined selected resources, 

and a set of presentation rules, which rules specify characteristics of the style of said 

generated presentation^ wherein said set of presentation rules are associated with said 

description scheme. 

25 

112. The method as claimed in claim 111, wherein said description scheme uses a 
declaration description definition language. 

113. Apparatus for generating on an output device a presentation based on a 
30 predetermined selection of resources, said apparatus comprising: 

means for reading a description scheme for said presentation, wherein the 
description scheme for said presentation uses a declarative description definition 
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language which contains definitions for descriptor components of the description 
scheme; 

means for generating a description of the said presentation using said 
description scheme and said predetermined selection of said resources; and 
5 means for generating on said output device the said presentation based on the 

said description of the said presentation and the predetermined selected resources, 

114, Apparatus for generating on an output device a presentation based on a 
predetermined selection of resources, said apparatus comprising: 

10 means for reading a description scheme for said presentation, wherein said 

description scheme contains definitions for descriptor components of the description 
scheme, and each said descriptor component comprises the association of a 
presentation attribute with a representative value for that attribute; 

means for generating a description of the said presentation using said 

15 description scheme and said predetermined selection of resources; and 

means for generating on said output device the said presentation based on the 
said generated description of the said presentation, the predetermined selected 
resources, and a set of presentation rules, which rules specify characteristics of the 
style of said generated presentation, wherein said set of presentation rules are 

20 associated with said description scheme. 

115. A computer readable medium comprising a computer program for generating on an 
output device a presentation based on a predetermined selection of resources,, said 
computer program comprising: 

25 code for reading a description scheme for said presentation, wherein the 

description scheme for said presentation uses a declarative description definition 
language which contains definitions for descriptor components of the description 
scheme; 

code for generating a description of the said presentation using said 
30 description scheme and said predetermined selection of said resources; and 

code for generating on said output device the said presentation based on the 
said description of the said presentation and the predetermined selected resources. 

CFP J 595US IPR34-38J3RP1490568US I:\ELECNC1SRA\IPRMPR34-38.GRP1 \40O5C8US.doc :PWM, 



-157- 



116. A computer readable medium comprising a computer program for generating on an 
output device a presentation based on a predetermined selection of resources, said 
computer program comprising: 

code for reading a description scheme for said presentation, wherein said 
description scheme contains definitions for descriptor components of the description 
scheme, and each said descriptor component comprises the association of a 
presentation attribute with a representative value for that attribute; 

code for generating a description of the said presentation using said 
description scheme and said predetermined selection of resources; and 

code for generating on said output device the said presentation based on the 
said generated description of the said presentation, the predetermined selected 
resources, and a set of presentation rules, which rules specify characteristics of the 
style of said generated presentation, wherein said set of presentation rules are 
associated with said description scheme. 
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-1 SB- 
ABSTRACT 



Browsing Electronically-Accessible Resources 



5 A method of browsing electronically-accessible resources using descriptions of the 

resources. These descriptions have descriptor components, which have attributes 
representative of at least two axes of access to the resources. These descriptions also have 
links to the corresponding resources. The method first reads the descriptions and displays 
a number of items (1608). Each one of these items are associated with a corresponding 
10 descriptor component of the read description that has an attribute. The method allows the 
browsing (1602,1603) of the descriptions and their corresponding electronically- 
accessible resources via the links using the displayed items. 
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